Skip to content

Document news digest methodology#4183

Merged
koala73 merged 3 commits into
mainfrom
codex/news-digest-methodology
Jun 7, 2026
Merged

Document news digest methodology#4183
koala73 merged 3 commits into
mainfrom
codex/news-digest-methodology

Conversation

@koala73

@koala73 koala73 commented Jun 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a canonical public methodology page for the news digest and briefing pipeline
  • link the page from news feed and signal-intelligence docs navigation
  • update NewsService proto/OpenAPI comments and add a focused methodology parity test
  • tighten story-track/freshness documentation comments after review

Validation

  • make generate
  • npm_config_cache=/tmp/worldmonitor-npm-cache npx markdownlint-cli2 docs/methodology/news-digest-and-briefing.mdx docs/panels/news-feeds.mdx docs/signal-intelligence.mdx
  • npm_config_cache=/tmp/worldmonitor-npm-cache npx tsx --test tests/news-digest-methodology-parity.test.mjs tests/news-freshness-floor.test.mts tests/news-classifier-historical-downgrade.test.mts tests/news-rss-description-extract.test.mts tests/news-story-track-description-persistence.test.mts tests/digest-buildDigest-feelgood-filter.test.mjs tests/digest-buildDigest-ephemeral-live-filter.test.mjs tests/digest-cooldown-config.test.mjs tests/digest-cooldown-decision.test.mjs tests/importance-score-parity.test.mjs tests/brief-dedup-embedding.test.mjs tests/brief-filter.test.mjs tests/brief-from-digest-stories.test.mjs
  • git diff --check
  • pre-push hook: typecheck, API typecheck, Convex typecheck, CJS syntax, Unicode safety, boundary lint, safe HTML, Sentry coverage, rate-limit policies, premium-fetch parity, edge bundle check, server handler tests, changed test files, markdown lint, MDX lint, proto freshness, pro-test freshness, version sync

@mintlify

mintlify Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
WorldMonitor 🟢 Ready View Preview Jun 7, 2026, 2:51 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@vercel

vercel Bot commented Jun 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment Jun 7, 2026 3:34am

Request Review

@greptile-apps

greptile-apps Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a canonical public methodology page for the news digest and briefing pipeline, links it from two existing doc pages, corrects a stale 48h96h comment in list-feed-digest.ts, and extends proto/OpenAPI comments with the commodity variant and two new feedStatuses values (all-undated, partial-undated). A new focused parity test locks the public constants that API consumers and documentation readers rely on.

  • New methodology doc (docs/methodology/news-digest-and-briefing.mdx): covers feed inventory, freshness floor, classification caps, importance-score formula, story-track Redis schema, digest/brief read path, cooldown table, LLM grounding, and bias posture — all cross-checked by the parity test.
  • Proto and OpenAPI updates: commodity variant added to ListFeedDigestRequest description; all-undated and partial-undated status codes added to feedStatuses comment, now consistent across .proto, .yaml, and .json artefacts.
  • Parity test (tests/news-digest-methodology-parity.test.mjs): text-scans implementation files for key constants and confirms they are documented; two extraction regexes have minor fragility (see inline comments).

Confidence Score: 4/5

Safe to merge — all runtime behaviour is unchanged; every code file touched contains only comment or documentation updates.

The only executable change is correcting a stale inline comment in list-feed-digest.ts (48h to 96h); the actual resolveMaxAgeMs() default was already 96h. The new parity test has two moderately fragile regex patterns that could produce misleading failure messages on future refactors, but neither masks a real bug today.

tests/news-digest-methodology-parity.test.mjs — the MAX_STORIES_PER_USER and cooldown-modes extraction patterns are worth tightening before the test surface grows.

Important Files Changed

Filename Overview
docs/methodology/news-digest-and-briefing.mdx New canonical methodology page covering feed inventory, freshness floor, classification, importance scoring, story tracking, brief read path, cooldowns, LLM usage, and bias posture. Content is internally consistent and aligns with the parity test assertions.
tests/news-digest-methodology-parity.test.mjs New focused parity test that guards public constants against methodology drift; two regex patterns have minor fragility risks (MAX_STORIES_PER_USER order dependency, cooldown modes requiring exactly 2-element Set literals).
server/_shared/cache-keys.ts Added publishedAt and entityCorroborationCount to both comment blocks; peakScore is missing from the inline JSDoc slash-list but present in the multi-line comment header.
server/worldmonitor/news/v1/list-feed-digest.ts Comment-only fix correcting the NEWS_MAX_AGE_HOURS default from 48h to 96h to match the actual resolveMaxAgeMs() implementation.
proto/worldmonitor/news/v1/list_feed_digest.proto Added commodity variant and two new feed-status values (all-undated, partial-undated) to proto comments; consistent with OpenAPI and methodology doc updates.
docs/api/NewsService.openapi.yaml Variant description updated to include commodity and feed-status values expanded with all-undated and partial-undated; mirrors proto changes.
docs/docs.json Wires the new methodology/news-digest-and-briefing page into the docs navigation under the methodology section.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[RSS/Atom Fetch
direct to relay fallback] --> B[Parse items
max 5 per feed]
    B --> C{Date valid?
not future > 1h}
    C -- drop --> D[feedStatus:
all-undated / partial-undated]
    C -- keep --> E[Freshness floor
NEWS_MAX_AGE_HOURS default 96h]
    E -- stale drop --> F[feedStatus: empty]
    E -- fresh --> G[Keyword Classifier
threat + category + confidence]
    G --> H{Historical marker?}
    H -- yes --> I[Downgrade to info]
    H -- no --> J[LLM classify-cache
capped at +2 levels]
    I --> K[Importance Score
severity x0.55 + tier x0.20
+ corroboration x0.15 + recency x0.10]
    J --> K
    K --> L[Sort and cap
20 per category]
    L --> M[Redis story-track write
story:track:v1 7d TTL
digest:accumulator:v1 48h TTL]
    M --> N[Digest cron
read accumulator window]
    N --> O[Read-path filters
opinion, feel-good,
ephemeral-live, fading]
    O --> P[Embedding dedup
cosine 0.60 / Jaccard fallback]
    P --> Q[Pool cap 30 clusters
high le 15, medium le 10]
    Q --> R[LLM brief prose
+ per-story whyMatters]
    R --> S[WorldMonitor Brief
max 12 stories
2 per source/category]
Loading

Comments Outside Diff (2)

  1. tests/news-digest-methodology-parity.test.mjs, line 692-695 (link)

    P2 Order-dependent MAX_STORIES_PER_USER check may give a false positive. The regex /return\s+12\s*;[\s\S]*export\s+const\s+MAX_STORIES_PER_USER/s passes if any return 12; appears anywhere in the file before the export declaration — including an unrelated function. If MAX_STORIES_PER_USER is later declared near the top of the module (e.g., during a refactor), this assertion could start failing despite a correct default. A tighter pattern anchoring to the actual default expression would be more robust.

  2. tests/news-digest-methodology-parity.test.mjs, line 751-754 (link)

    P2 Cooldown mode extraction silently produces an empty array if the Set literal is reformatted. The regex requires exactly two items with no trailing comma. If a trailing comma is added or the array is expanded to three items, modes becomes [] and the deepEqual fails with [] !== ['shadow', 'off'] — correct behavior, but the root cause (regex mismatch) may not be obvious to the next developer. A simpler extraction not tied to the literal Set syntax would be more resilient.

Reviews (1): Last reviewed commit: "docs(news): document digest methodology" | Re-trigger Greptile

Comment thread server/_shared/cache-keys.ts Outdated
@koala73
koala73 merged commit 0983d52 into main Jun 7, 2026
18 checks passed
@koala73
koala73 deleted the codex/news-digest-methodology branch June 7, 2026 03:46
Yashraghuvans pushed a commit to Yashraghuvans/worldmonitor that referenced this pull request Jun 7, 2026
…ala73#4183 is unresolved (koala73#3841)

Upstream bug: `umami-software/umami#4183` (open since 2026-04-19, OP and a
second reporter both running self-hosted v3.1.0 against Postgres). A race
in `prisma.sessionData.updateMany()` returns HTTP 500 from `/api/send`
with `Unique constraint failed on the fields: (session_data_id)` for
~4-8% of requests. Maintainer acknowledged 2026-04-22; suggested fix is
to add a `(sessionId, dataKey)` composite unique and switch to `upsert`,
but no commit has landed on master (22 commits since 3.1.0 — all
dependency bumps + a chart canvas tweak, none touch sessionData).

Self-hosted on Railway runs `umamisoftware/umami:3.1.0` against
`postgres.railway.internal` — Railway service source pinned to that
tag in a separate API change so the rolling `postgresql-latest` can't
drift back onto the same broken digest.

Until a tagged release ships the fix, drop the three variants currently
reporting 500-storm symptoms from `data-domains` in `index.html:121`:
`tech.worldmonitor.app`, `finance.worldmonitor.app`,
`commodity.worldmonitor.app`. Umami's tracker self-disables when the
current hostname isn't in `data-domains` — same mechanism that's been
keeping `energy.worldmonitor.app` quiet (it was never added).

Kept active intentionally:
- `worldmonitor.app` (main host, residual analytics signal)
- `happy.worldmonitor.app` (low-traffic, near-zero 500 volume)

If you want the Grafana error-rate metric flat at 0%, drop those too —
this PR preserves them deliberately so the analytics product isn't
fully blacked out for the ~days-to-weeks window before upstream ships
3.1.1 (or whatever tagged release contains koala73#4183's fix).

CSP `connect-src` / `script-src` still allow `abacus.worldmonitor.app`
because the script still loads on the kept hosts; only the data-domains
allowlist shrinks. No hash regeneration needed (the 4 sha256 hashes in
`script-src` cover inline scripts elsewhere in the document, not the
external Umami script which loads via origin allowlist).

To restore: add the three subdomains back to `data-domains` once
umami-software/umami releases a tag containing the sessionData race fix
(track via the linked issue).
koala73 added a commit that referenced this pull request Jun 22, 2026
…up tests

Code-review follow-ups for #4345:
- scripts/package.json: re-add overrides.undici "$undici" to match main
  (post-#4350). scripts/ is now byte-identical to main, clearing the husky
  pre-push lockfile-sync gate and re-pinning transitive undici. Lockfile is
  unchanged (already dedupes to 7.28.0).
- analytics.ts: fix the loadUmamiScript existing-script branch (flush only
  when window.umami is set, else attach the load listener; set
  umamiLoadStarted to close the re-entry gap); make initAnalytics synchronous
  (no await, sole caller voids it); add resetAnalyticsForTesting; move the
  #4183 data-domains note next to UMAMI_DOMAINS as the single source of truth.
- index.html: trim the Umami comment to a pointer at the new source of truth.
- tests: add coverage for scheduleAfterFirstPaint load-listener + rIC-absent
  fallback branches, Umami retry-exhaustion, 50-cap queue overflow,
  throw-then-queue-then-flush, and the existing-script branch.
- e2e: replace the flaky 250ms wait with the deterministic
  wmEventHandlersReady readiness gate.

Claude-Session: https://claude.ai/code/session_015dspobi7rUanvd9zSamz8a
koala73 added a commit that referenced this pull request Jun 22, 2026
* perf(dashboard): defer secondary startup payloads

* fix(security): bump scripts undici

* test(dashboard): assert deferred analytics startup

* fix(analytics): queue umami events on throw + restore happy Nunito italic

- sendUmamiCall: return false in the catch so a throwing window.umami
  defers to the bounded queue/flush path instead of silently dropping the
  event (callers only queue when sendUmamiCall returns false).
- buildDashboardFontStylesheetHref: restore the Nunito italic axis
  (ital,wght@0,400;0,600;0,700;1,400) the narrowed request dropped;
  happy-theme.css uses font-style:italic on Nunito-bodied elements, so the
  weight-only request regressed those to synthesized oblique. Weight
  narrowing (no 300) preserved.
- Update the two font-href assertions that had locked in the regression.

Addresses ce-code-review findings on PR #4345.

Claude-Session: https://claude.ai/code/session_0187W7sB8cdp8sYK7J7qDNLn

* fix(analytics): align scripts undici override + harden deferred-startup tests

Code-review follow-ups for #4345:
- scripts/package.json: re-add overrides.undici "$undici" to match main
  (post-#4350). scripts/ is now byte-identical to main, clearing the husky
  pre-push lockfile-sync gate and re-pinning transitive undici. Lockfile is
  unchanged (already dedupes to 7.28.0).
- analytics.ts: fix the loadUmamiScript existing-script branch (flush only
  when window.umami is set, else attach the load listener; set
  umamiLoadStarted to close the re-entry gap); make initAnalytics synchronous
  (no await, sole caller voids it); add resetAnalyticsForTesting; move the
  #4183 data-domains note next to UMAMI_DOMAINS as the single source of truth.
- index.html: trim the Umami comment to a pointer at the new source of truth.
- tests: add coverage for scheduleAfterFirstPaint load-listener + rIC-absent
  fallback branches, Umami retry-exhaustion, 50-cap queue overflow,
  throw-then-queue-then-flush, and the existing-script branch.
- e2e: replace the flaky 250ms wait with the deterministic
  wmEventHandlersReady readiness gate.

Claude-Session: https://claude.ai/code/session_015dspobi7rUanvd9zSamz8a
koala73 added a commit that referenced this pull request Jul 6, 2026
…ckout funnel events (#4931) (#4934)

* feat(analytics): restore Umami on canonical www host + end-to-end checkout funnel events (#4931)

The conversion funnel was unmeasurable and partially dark:

- UMAMI_DOMAINS listed only the apex, but the apex 301s to
  www.worldmonitor.app and the tracker's data-domains check is an EXACT
  hostname match -- every dashboard event on the canonical host was
  silently dropped. www is now listed (tech/finance/commodity stay
  excluded per the upstream Umami #4183 note).
- New typed funnel events: checkout-start (fired from startCheckout on
  the dashboard and from the /pro checkout service, with surface +
  authed props), checkout-success / checkout-failed (fired from the
  checkout-return reconciliation in panel-layout).
- The /pro SPA and welcome landing now load the tracker (static
  wm-static-bootstrap nonce satisfies the strict-dynamic CSP) and the
  primary welcome CTAs carry data-umami-event attributes matching their
  existing ?ref= markers.
- public/pro rebuilt; funnel-analytics-policy test guards the domains
  list, catalog entries, call sites, and tracker tags against silent
  removal.

Closes #4931

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s

* test(checkout): stub ./analytics in the esbuild checkout harnesses

checkout.ts now statically imports trackCheckoutStart (#4931); the
overlay-lifecycle and pending-dialog harnesses stub ./billing and
./clerk with only the exports checkout.ts itself uses, so the real
analytics module (which imports onSubscriptionChange and
getClerkUserCreatedAt from those modules) failed the bundle. A no-op
./analytics stub keeps the facade out of the bundle; its behavior is
covered by tests/secondary-startup.test.mts.

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s

* fix(analytics): happy-host tracker coverage + distinct dashboard-resume surface (PR #4934 review)

Review findings from the adversarial pass (verdict: nothing blocking;
two LOWs fixed for completeness):

- happy.worldmonitor.app rewrites / to the welcome landing, but the pro
  pages' data-domains omitted it -- the landing/pro funnel was blind on
  that host while the dashboard tracked. Added to both pro HTML files.
- The post-sign-in auto-resume re-entered startCheckout and fired a
  second checkout-start with the same 'dashboard' surface, reading a
  signed-out conversion as two attempts. The resume now tags
  surface:'dashboard-resume' (mirroring /pro's pro-page/pro-resume
  split) so funnel queries can dedupe cleanly.

public/pro rebuilt.

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s

* fix(analytics): async tracker + param hygiene, durable checkout-success, closed failure buckets (PR #4934 review round 2)

Four P2 findings, all fixed:

- F1: the tracker tags are now async (a plain defer script holds
  DOMContentLoaded hostage to the analytics host), and the /pro SPA tag
  adds data-exclude-search so checkout-intent params (wm_checkout_*
  product/discount/referral) and Clerk handshake params never land in
  analytics. The welcome landing keeps query tracking (utm/?lang=
  segmentation; no sensitive params are ever routed to /).
- F2: checkout-success is the one funnel event that races a reload --
  the entitlement watcher reloads the dashboard the moment Pro lands,
  often before the deferred Umami queue flushes. trackCheckoutSuccess
  now writes a per-tab sessionStorage marker that is cleared only on
  actual delivery (sendUmamiCall); panel-layout replays it on the next
  boot (replayed:true). Behavioral test covers deliver-clears / replay /
  no-double-count.
- F3: checkout-failed forwards a URL-derived status; checkout-return
  deliberately passes unknown statuses through when Dodo ID params are
  present, so a crafted URL could inject unbounded cardinality.
  Normalized to the closed failed/declined/cancelled/canceled/other set.
- F4 (Greptile): the /pro policy test asserted only the surface labels,
  not that trackFunnelEvent('checkout-start') is actually emitted for
  each path -- tightened to extract emissions, plus new policy guards
  for async/exclude-search, the durable marker wiring, and the closed
  failure vocabulary.

public/pro rebuilt; 33/33 across the durable, policy, secondary-startup
and checkout-harness suites; typecheck clean.

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s

* fix(analytics): product-id bucketing, /pro funnel queue, whole-start guard (PR #4934 review round 4)

Elie's cross-PR review, three P2s:

- F2 cardinality: checkout-start productId travels through URL
  (wm_checkout_product) and sessionStorage on the resume paths, so a
  crafted value could inject unbounded cardinality into Umami. Both
  surfaces now bucket unknown ids to 'unknown' against generated
  allowlists (dashboard: DODO_PRODUCTS from products.generated.ts;
  /pro: tiers.json product ids -- exactly the purchasable set per
  build). Checkout itself still passes the raw id through; the backend
  validates. Behavioral test asserts crafted-id collapse + known-id
  passthrough.
- F3 dropped resume event: the /pro tracker is async and typically
  loads AFTER the mount-time tryResumeCheckoutFromUrl, so the silent
  no-op trackFunnelEvent dropped the only signed-in pro-resume event.
  /pro funnel events now queue (bounded, 20) and flush via a 500ms
  poll for up to 30s once window.umami appears.
- F4 double-fire: checkoutInFlight is only set inside doCheckout,
  after the awaited ensureClerk() -- rapid double-clicks both passed
  the entry check and fired checkout-start twice (checkout stayed
  single; analytics did not). A synchronous whole-start re-entrancy
  guard now wraps startCheckout.

Also fixed a latent policy-test bug the bucketing exposed: the
emission-extraction regex stopped at the FIRST ')' -- which is now the
inner bucketProductIdForAnalytics() call -- so it truncated before the
surface label; it now spans to the closing '})'.

public/pro rebuilt; policy + durable + secondary-startup + both
checkout harnesses green; main and pro tsc clean.

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s

* fix(analytics): persist /pro checkout-start across the Dodo redirect (PR #4934 review round 5)

The round-4 in-memory queue never got to flush on the fast path: a
signed-in click (or auto-resume) continues straight into doCheckout's
top-level redirect to Dodo, unloading /pro with the event still queued.

- /pro now mirrors undelivered checkout-start events into sessionStorage
  (per-tab, survives the Dodo round-trip; bounded at 10) and clears the
  mirror the moment a local flush delivers, so nothing double-replays.
- The dashboard return landing replays them unconditionally at boot
  (both checkout-return and ordinary branches -- the buyer can land on
  either). Every replayed field is re-validated against closed
  vocabularies (event name must be checkout-start, productId re-bucketed
  via the generated allowlist, surface coerced to pro-page/pro-resume,
  authed coerced to boolean) because sessionStorage is client-writable
  and replayed junk must not become analytics cardinality; replays carry
  replayed:true.
- Behavioral tests: valid event replays sanitized, crafted fields
  collapse, unknown event names and garbage entries drop, malformed
  storage clears without tracking, ordinary boots are no-ops. Policy
  test pins the handoff key in BOTH builds + the persist/clear/replay
  wiring.

public/pro rebuilt; policy + durable + secondary-startup + both
checkout harnesses green; main and pro tsc clean.

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s

* fix(analytics): clear the /pro replay marker on delivery, not on read (PR #4934 review round 6)

The round-5 replay removed wm-pro-funnel-pending at READ time, then the
replayed events sat in the deferred in-memory queue -- the entitlement
watcher reload (the exact race round-2 fixed for checkout-success)
could kill the queue before Umami loaded, permanently dropping the
/pro checkout-start with the durable handoff already gone.

Now mirrors the checkout-success contract exactly:
- replayPendingProFunnelEvents no longer removes the key; it REWRITES
  it with only the sanitized survivors (closed vocabularies, bounded)
  so a pre-delivery reload retries exactly those and junk cannot loop,
  clearing immediately only when nothing is replayable (malformed /
  all-invalid payloads).
- sendUmamiCall clears the marker once a replayed checkout-start
  (data.replayed === true) actually reaches the tracker. Live
  dashboard checkout-starts never clear it. All replays flush in one
  synchronous loop, so first-delivery-clears is safe.
- Tests now encode the delivery contract: marker persists (sanitized)
  through a pre-delivery reload and the retry delivers; clears on
  delivery; unreplayable payloads still clear. The old assertion that
  pinned clear-on-read is gone. Policy test pins the delivery hook and
  the sanitized rewrite.

Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant