Document news digest methodology#4183
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds a canonical public methodology page for the news digest and briefing pipeline, links it from two existing doc pages, corrects a stale
Confidence Score: 4/5Safe to merge — all runtime behaviour is unchanged; every code file touched contains only comment or documentation updates. The only executable change is correcting a stale inline comment in list-feed-digest.ts (48h to 96h); the actual resolveMaxAgeMs() default was already 96h. The new parity test has two moderately fragile regex patterns that could produce misleading failure messages on future refactors, but neither masks a real bug today. tests/news-digest-methodology-parity.test.mjs — the MAX_STORIES_PER_USER and cooldown-modes extraction patterns are worth tightening before the test surface grows. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[RSS/Atom Fetch
direct to relay fallback] --> B[Parse items
max 5 per feed]
B --> C{Date valid?
not future > 1h}
C -- drop --> D[feedStatus:
all-undated / partial-undated]
C -- keep --> E[Freshness floor
NEWS_MAX_AGE_HOURS default 96h]
E -- stale drop --> F[feedStatus: empty]
E -- fresh --> G[Keyword Classifier
threat + category + confidence]
G --> H{Historical marker?}
H -- yes --> I[Downgrade to info]
H -- no --> J[LLM classify-cache
capped at +2 levels]
I --> K[Importance Score
severity x0.55 + tier x0.20
+ corroboration x0.15 + recency x0.10]
J --> K
K --> L[Sort and cap
20 per category]
L --> M[Redis story-track write
story:track:v1 7d TTL
digest:accumulator:v1 48h TTL]
M --> N[Digest cron
read accumulator window]
N --> O[Read-path filters
opinion, feel-good,
ephemeral-live, fading]
O --> P[Embedding dedup
cosine 0.60 / Jaccard fallback]
P --> Q[Pool cap 30 clusters
high le 15, medium le 10]
Q --> R[LLM brief prose
+ per-story whyMatters]
R --> S[WorldMonitor Brief
max 12 stories
2 per source/category]
|
…ala73#4183 is unresolved (koala73#3841) Upstream bug: `umami-software/umami#4183` (open since 2026-04-19, OP and a second reporter both running self-hosted v3.1.0 against Postgres). A race in `prisma.sessionData.updateMany()` returns HTTP 500 from `/api/send` with `Unique constraint failed on the fields: (session_data_id)` for ~4-8% of requests. Maintainer acknowledged 2026-04-22; suggested fix is to add a `(sessionId, dataKey)` composite unique and switch to `upsert`, but no commit has landed on master (22 commits since 3.1.0 — all dependency bumps + a chart canvas tweak, none touch sessionData). Self-hosted on Railway runs `umamisoftware/umami:3.1.0` against `postgres.railway.internal` — Railway service source pinned to that tag in a separate API change so the rolling `postgresql-latest` can't drift back onto the same broken digest. Until a tagged release ships the fix, drop the three variants currently reporting 500-storm symptoms from `data-domains` in `index.html:121`: `tech.worldmonitor.app`, `finance.worldmonitor.app`, `commodity.worldmonitor.app`. Umami's tracker self-disables when the current hostname isn't in `data-domains` — same mechanism that's been keeping `energy.worldmonitor.app` quiet (it was never added). Kept active intentionally: - `worldmonitor.app` (main host, residual analytics signal) - `happy.worldmonitor.app` (low-traffic, near-zero 500 volume) If you want the Grafana error-rate metric flat at 0%, drop those too — this PR preserves them deliberately so the analytics product isn't fully blacked out for the ~days-to-weeks window before upstream ships 3.1.1 (or whatever tagged release contains koala73#4183's fix). CSP `connect-src` / `script-src` still allow `abacus.worldmonitor.app` because the script still loads on the kept hosts; only the data-domains allowlist shrinks. No hash regeneration needed (the 4 sha256 hashes in `script-src` cover inline scripts elsewhere in the document, not the external Umami script which loads via origin allowlist). To restore: add the three subdomains back to `data-domains` once umami-software/umami releases a tag containing the sessionData race fix (track via the linked issue).
…up tests Code-review follow-ups for #4345: - scripts/package.json: re-add overrides.undici "$undici" to match main (post-#4350). scripts/ is now byte-identical to main, clearing the husky pre-push lockfile-sync gate and re-pinning transitive undici. Lockfile is unchanged (already dedupes to 7.28.0). - analytics.ts: fix the loadUmamiScript existing-script branch (flush only when window.umami is set, else attach the load listener; set umamiLoadStarted to close the re-entry gap); make initAnalytics synchronous (no await, sole caller voids it); add resetAnalyticsForTesting; move the #4183 data-domains note next to UMAMI_DOMAINS as the single source of truth. - index.html: trim the Umami comment to a pointer at the new source of truth. - tests: add coverage for scheduleAfterFirstPaint load-listener + rIC-absent fallback branches, Umami retry-exhaustion, 50-cap queue overflow, throw-then-queue-then-flush, and the existing-script branch. - e2e: replace the flaky 250ms wait with the deterministic wmEventHandlersReady readiness gate. Claude-Session: https://claude.ai/code/session_015dspobi7rUanvd9zSamz8a
* perf(dashboard): defer secondary startup payloads * fix(security): bump scripts undici * test(dashboard): assert deferred analytics startup * fix(analytics): queue umami events on throw + restore happy Nunito italic - sendUmamiCall: return false in the catch so a throwing window.umami defers to the bounded queue/flush path instead of silently dropping the event (callers only queue when sendUmamiCall returns false). - buildDashboardFontStylesheetHref: restore the Nunito italic axis (ital,wght@0,400;0,600;0,700;1,400) the narrowed request dropped; happy-theme.css uses font-style:italic on Nunito-bodied elements, so the weight-only request regressed those to synthesized oblique. Weight narrowing (no 300) preserved. - Update the two font-href assertions that had locked in the regression. Addresses ce-code-review findings on PR #4345. Claude-Session: https://claude.ai/code/session_0187W7sB8cdp8sYK7J7qDNLn * fix(analytics): align scripts undici override + harden deferred-startup tests Code-review follow-ups for #4345: - scripts/package.json: re-add overrides.undici "$undici" to match main (post-#4350). scripts/ is now byte-identical to main, clearing the husky pre-push lockfile-sync gate and re-pinning transitive undici. Lockfile is unchanged (already dedupes to 7.28.0). - analytics.ts: fix the loadUmamiScript existing-script branch (flush only when window.umami is set, else attach the load listener; set umamiLoadStarted to close the re-entry gap); make initAnalytics synchronous (no await, sole caller voids it); add resetAnalyticsForTesting; move the #4183 data-domains note next to UMAMI_DOMAINS as the single source of truth. - index.html: trim the Umami comment to a pointer at the new source of truth. - tests: add coverage for scheduleAfterFirstPaint load-listener + rIC-absent fallback branches, Umami retry-exhaustion, 50-cap queue overflow, throw-then-queue-then-flush, and the existing-script branch. - e2e: replace the flaky 250ms wait with the deterministic wmEventHandlersReady readiness gate. Claude-Session: https://claude.ai/code/session_015dspobi7rUanvd9zSamz8a
…ckout funnel events (#4931) (#4934) * feat(analytics): restore Umami on canonical www host + end-to-end checkout funnel events (#4931) The conversion funnel was unmeasurable and partially dark: - UMAMI_DOMAINS listed only the apex, but the apex 301s to www.worldmonitor.app and the tracker's data-domains check is an EXACT hostname match -- every dashboard event on the canonical host was silently dropped. www is now listed (tech/finance/commodity stay excluded per the upstream Umami #4183 note). - New typed funnel events: checkout-start (fired from startCheckout on the dashboard and from the /pro checkout service, with surface + authed props), checkout-success / checkout-failed (fired from the checkout-return reconciliation in panel-layout). - The /pro SPA and welcome landing now load the tracker (static wm-static-bootstrap nonce satisfies the strict-dynamic CSP) and the primary welcome CTAs carry data-umami-event attributes matching their existing ?ref= markers. - public/pro rebuilt; funnel-analytics-policy test guards the domains list, catalog entries, call sites, and tracker tags against silent removal. Closes #4931 Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s * test(checkout): stub ./analytics in the esbuild checkout harnesses checkout.ts now statically imports trackCheckoutStart (#4931); the overlay-lifecycle and pending-dialog harnesses stub ./billing and ./clerk with only the exports checkout.ts itself uses, so the real analytics module (which imports onSubscriptionChange and getClerkUserCreatedAt from those modules) failed the bundle. A no-op ./analytics stub keeps the facade out of the bundle; its behavior is covered by tests/secondary-startup.test.mts. Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s * fix(analytics): happy-host tracker coverage + distinct dashboard-resume surface (PR #4934 review) Review findings from the adversarial pass (verdict: nothing blocking; two LOWs fixed for completeness): - happy.worldmonitor.app rewrites / to the welcome landing, but the pro pages' data-domains omitted it -- the landing/pro funnel was blind on that host while the dashboard tracked. Added to both pro HTML files. - The post-sign-in auto-resume re-entered startCheckout and fired a second checkout-start with the same 'dashboard' surface, reading a signed-out conversion as two attempts. The resume now tags surface:'dashboard-resume' (mirroring /pro's pro-page/pro-resume split) so funnel queries can dedupe cleanly. public/pro rebuilt. Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s * fix(analytics): async tracker + param hygiene, durable checkout-success, closed failure buckets (PR #4934 review round 2) Four P2 findings, all fixed: - F1: the tracker tags are now async (a plain defer script holds DOMContentLoaded hostage to the analytics host), and the /pro SPA tag adds data-exclude-search so checkout-intent params (wm_checkout_* product/discount/referral) and Clerk handshake params never land in analytics. The welcome landing keeps query tracking (utm/?lang= segmentation; no sensitive params are ever routed to /). - F2: checkout-success is the one funnel event that races a reload -- the entitlement watcher reloads the dashboard the moment Pro lands, often before the deferred Umami queue flushes. trackCheckoutSuccess now writes a per-tab sessionStorage marker that is cleared only on actual delivery (sendUmamiCall); panel-layout replays it on the next boot (replayed:true). Behavioral test covers deliver-clears / replay / no-double-count. - F3: checkout-failed forwards a URL-derived status; checkout-return deliberately passes unknown statuses through when Dodo ID params are present, so a crafted URL could inject unbounded cardinality. Normalized to the closed failed/declined/cancelled/canceled/other set. - F4 (Greptile): the /pro policy test asserted only the surface labels, not that trackFunnelEvent('checkout-start') is actually emitted for each path -- tightened to extract emissions, plus new policy guards for async/exclude-search, the durable marker wiring, and the closed failure vocabulary. public/pro rebuilt; 33/33 across the durable, policy, secondary-startup and checkout-harness suites; typecheck clean. Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s * fix(analytics): product-id bucketing, /pro funnel queue, whole-start guard (PR #4934 review round 4) Elie's cross-PR review, three P2s: - F2 cardinality: checkout-start productId travels through URL (wm_checkout_product) and sessionStorage on the resume paths, so a crafted value could inject unbounded cardinality into Umami. Both surfaces now bucket unknown ids to 'unknown' against generated allowlists (dashboard: DODO_PRODUCTS from products.generated.ts; /pro: tiers.json product ids -- exactly the purchasable set per build). Checkout itself still passes the raw id through; the backend validates. Behavioral test asserts crafted-id collapse + known-id passthrough. - F3 dropped resume event: the /pro tracker is async and typically loads AFTER the mount-time tryResumeCheckoutFromUrl, so the silent no-op trackFunnelEvent dropped the only signed-in pro-resume event. /pro funnel events now queue (bounded, 20) and flush via a 500ms poll for up to 30s once window.umami appears. - F4 double-fire: checkoutInFlight is only set inside doCheckout, after the awaited ensureClerk() -- rapid double-clicks both passed the entry check and fired checkout-start twice (checkout stayed single; analytics did not). A synchronous whole-start re-entrancy guard now wraps startCheckout. Also fixed a latent policy-test bug the bucketing exposed: the emission-extraction regex stopped at the FIRST ')' -- which is now the inner bucketProductIdForAnalytics() call -- so it truncated before the surface label; it now spans to the closing '})'. public/pro rebuilt; policy + durable + secondary-startup + both checkout harnesses green; main and pro tsc clean. Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s * fix(analytics): persist /pro checkout-start across the Dodo redirect (PR #4934 review round 5) The round-4 in-memory queue never got to flush on the fast path: a signed-in click (or auto-resume) continues straight into doCheckout's top-level redirect to Dodo, unloading /pro with the event still queued. - /pro now mirrors undelivered checkout-start events into sessionStorage (per-tab, survives the Dodo round-trip; bounded at 10) and clears the mirror the moment a local flush delivers, so nothing double-replays. - The dashboard return landing replays them unconditionally at boot (both checkout-return and ordinary branches -- the buyer can land on either). Every replayed field is re-validated against closed vocabularies (event name must be checkout-start, productId re-bucketed via the generated allowlist, surface coerced to pro-page/pro-resume, authed coerced to boolean) because sessionStorage is client-writable and replayed junk must not become analytics cardinality; replays carry replayed:true. - Behavioral tests: valid event replays sanitized, crafted fields collapse, unknown event names and garbage entries drop, malformed storage clears without tracking, ordinary boots are no-ops. Policy test pins the handoff key in BOTH builds + the persist/clear/replay wiring. public/pro rebuilt; policy + durable + secondary-startup + both checkout harnesses green; main and pro tsc clean. Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s * fix(analytics): clear the /pro replay marker on delivery, not on read (PR #4934 review round 6) The round-5 replay removed wm-pro-funnel-pending at READ time, then the replayed events sat in the deferred in-memory queue -- the entitlement watcher reload (the exact race round-2 fixed for checkout-success) could kill the queue before Umami loaded, permanently dropping the /pro checkout-start with the durable handoff already gone. Now mirrors the checkout-success contract exactly: - replayPendingProFunnelEvents no longer removes the key; it REWRITES it with only the sanitized survivors (closed vocabularies, bounded) so a pre-delivery reload retries exactly those and junk cannot loop, clearing immediately only when nothing is replayable (malformed / all-invalid payloads). - sendUmamiCall clears the marker once a replayed checkout-start (data.replayed === true) actually reaches the tracker. Live dashboard checkout-starts never clear it. All replays flush in one synchronous loop, so first-delivery-clears is safe. - Tests now encode the delivery contract: marker persists (sanitized) through a pre-delivery reload and the retry delivers; clears on delivery; unreplayable payloads still clear. The old assertion that pinned clear-on-read is gone. Policy test pins the delivery hook and the sanitized rewrite. Claude-Session: https://claude.ai/code/session_0161Cso3K8pbtf276Q9WPM3s
Summary
Validation