fix(brief): single canonical synthesis — kill the compose/enrich/send parity regression#3685
Conversation
… parity regression The May 14 0802 brief tripped `PARITY REGRESSION` for a real user: the email channel lead (403 chars) and the magazine envelope lead (373 chars) were different syntheses. The canonical-brain promise from PR #3396 ("one synthesis, every channel byte-identical") had regressed — there were THREE synthesis call sites, not one: 1. Compose pass — runSynthesisWithFallback (seed-digest:1530), ctx-aware, spliced into the envelope. Its output never survived. 2. enrichBriefEnvelopeWithLLM — generateDigestProse (brief-llm.mjs) with a 4-arg NO-ctx call that OVERWROTE envelope.data.digest. The comment claimed "this pass only fills per-story rationales"; it lied — it re-synthesised the whole digest prose. 3. Send pass — runSynthesisWithFallback (seed-digest:1949) against a different `stories` pool, for the channel bodies. So compose built a ctx-aware synthesis, enrich threw it away and re-rolled context-free, and send rolled a third time. At temperature 0.4 the three rolls diverge. Fix — collapse to ONE synthesis, produced by the compose pass: - enrichBriefEnvelopeWithLLM gains `opts.skipDigestProse`. When set, it does ONLY per-story whyMatters/description enrichment and leaves envelope.data.digest untouched — no second generateDigestProse call. The compose path passes skipDigestProse:true (the canonical synthesis is already spliced in). Call site 2 is gone. - composeAndStoreBriefForUser now returns the `synthesis` object on its briefByUser entry (alongside the existing synthesisLevel). - The send pass no longer calls runSynthesisWithFallback. It reads briefSynthesis / briefLead / synthesisLevel from the compose-pass result on the briefByUser entry. Call site 3 is gone. - L3 / opt-out gate preserved: briefSynthesis/briefLead are only set when AI_DIGEST_ENABLED && rule.aiDigestEnabled !== false && synthesisLevel !== 3. The persisted envelope always carries a digest.lead (the L3 stub included), so reading from the briefByUser entry — NOT the envelope — is what keeps L3/opt-out users from a fake "Executive Summary". - Parity log: channels_equal is now `n/a` (not a misleading `false`) when there is no channel synthesis (L3 / opt-out / compose-miss), so those users stop tripping a false PARITY REGRESSION every tick. The remaining PARITY REGRESSION branch fires only on a genuine post-compose envelope mutation — which after this change should be unreachable for L1/L2. Legitimate non-violations preserved: generateDigestProsePublic stays (separate public/share synthesis), and the L1→L2 fallback inside runSynthesisWithFallback may still call generateDigestProse twice within one canonical attempt. Tests: 2 new enrichBriefEnvelopeWithLLM cases — skipDigestProse:true suppresses the generateDigestProse call + leaves digest untouched + still enriches whyMatters; default (no opts) still synthesises (back-compat). 93/93 brief-llm tests, 8702/8702 full unit suite, typecheck clean. Plan: docs/plans/2026-05-14-001-fix-brief-pipeline-parity-grounding-opinion-plan.md (F1, Phase 1). Codex-reviewed (3 rounds, APPROVED). Phase 1.5 (F5 root cause) finding: a harness proved orderBriefCandidates cannot produce the May 14 interleaved severity walk under any topic assignment — H2 (comparator bug) ruled out, H1 confirmed. The comparator is untouched; F5 gets a post-deploy verification step.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR collapses three independent synthesis call sites down to one canonical synthesis produced in the compose pass, eliminating the compose↔send parity regression that caused the email and magazine leads to diverge at temperature 0.4. It also fixes a false
Confidence Score: 4/5Safe to merge — the core parity fix is mechanically simple (pass synthesis through an in-memory Map instead of re-rolling it), tests are clean, and the trade-off for compose-miss users is explicitly documented and self-healing. The three-to-one synthesis consolidation is correct: the skipDigestProse flag prevents the compose-pass synthesis from being silently overwritten, and the send pass reading brief?.synthesis directly is simpler and more reliable than re-calling runSynthesisWithFallback. The channelsEqual === false guard is stricter than the old !channelsEqual && briefLead && envLead — it will now catch a mismatch even when envLead is an empty string, which is more correct but a subtle surface-area change worth watching post-deploy. The send-pass block in scripts/seed-digest-notifications.mjs around the briefSynthesis/briefLead assignment and the parity-log conditional — specifically the compose-miss fallback path where brief is undefined and synthesisLevel defaults to 3. Important Files Changed
Sequence DiagramsequenceDiagram
participant CP as Compose Pass
participant BU as briefByUser (in-memory)
participant EN as enrichBriefEnvelopeWithLLM
participant SP as Send Pass
CP->>CP: runSynthesisWithFallback(userId, winnerStories, ctx)
CP->>EN: enrichBriefEnvelopeWithLLM(envelope, rule, deps, skipDigestProse: true)
Note over EN: Skips generateDigestProse
EN-->>CP: enriched envelope (digest untouched)
CP->>BU: "briefByUser.set(userId, {envelope, synthesis, synthesisLevel})"
SP->>BU: "brief = briefByUser.get(userId)"
alt L1/L2 user
SP->>SP: "briefSynthesis = brief.synthesis"
Note over SP: channelsEqual = briefLead === envLead
else L3 / opt-out / compose-miss
SP->>SP: "briefLead = null"
Note over SP: channelsEqual = 'n/a'
end
SP->>SP: Render channel bodies using briefSynthesis
Reviews (1): Last reviewed commit: "fix(brief): single canonical synthesis —..." | Re-trigger Greptile |
…REGRESSION Review catch on PR #3685. The Phase-1 change fixed the channels_equal false-positive for L3/opt-out, but compose-miss still tripped the OTHER alarm: when briefByUser has no entry, `brief` is undefined, so the parity block computed winnerVariant='' → winnerMatch=false → fired `PARITY REGRESSION winner_match=false` every compose-miss tick. Split the parity block: when `!brief` (compose-miss), emit an informational parity line with winner_match=n/a channels_equal=n/a reason=compose-miss and skip BOTH alarms — the compose-miss is already logged separately via `[digest] compose-miss user=…`. The winner_match alarm now only fires when `brief` exists, which is the genuine bug case (canonical-rule filter bypass / chosenVariant drift). Updated the stale "unreachable in practice" comment to note it is reached only with a present `brief`. Pre-existing bug (compose-miss always set winnerVariant=''), but this PR owns the parity-log block and Phase 1 step 5 of the plan explicitly scopes "no compose-pass result → n/a, not a false regression" — compose-miss is the same family as L3/opt-out. typecheck clean, 143/143 brief-llm + digest-orchestration tests pass.
…+ restore the grounding gate (#3688) * fix(brief): single canonical synthesis — kill the compose/enrich/send parity regression The May 14 0802 brief tripped `PARITY REGRESSION` for a real user: the email channel lead (403 chars) and the magazine envelope lead (373 chars) were different syntheses. The canonical-brain promise from PR #3396 ("one synthesis, every channel byte-identical") had regressed — there were THREE synthesis call sites, not one: 1. Compose pass — runSynthesisWithFallback (seed-digest:1530), ctx-aware, spliced into the envelope. Its output never survived. 2. enrichBriefEnvelopeWithLLM — generateDigestProse (brief-llm.mjs) with a 4-arg NO-ctx call that OVERWROTE envelope.data.digest. The comment claimed "this pass only fills per-story rationales"; it lied — it re-synthesised the whole digest prose. 3. Send pass — runSynthesisWithFallback (seed-digest:1949) against a different `stories` pool, for the channel bodies. So compose built a ctx-aware synthesis, enrich threw it away and re-rolled context-free, and send rolled a third time. At temperature 0.4 the three rolls diverge. Fix — collapse to ONE synthesis, produced by the compose pass: - enrichBriefEnvelopeWithLLM gains `opts.skipDigestProse`. When set, it does ONLY per-story whyMatters/description enrichment and leaves envelope.data.digest untouched — no second generateDigestProse call. The compose path passes skipDigestProse:true (the canonical synthesis is already spliced in). Call site 2 is gone. - composeAndStoreBriefForUser now returns the `synthesis` object on its briefByUser entry (alongside the existing synthesisLevel). - The send pass no longer calls runSynthesisWithFallback. It reads briefSynthesis / briefLead / synthesisLevel from the compose-pass result on the briefByUser entry. Call site 3 is gone. - L3 / opt-out gate preserved: briefSynthesis/briefLead are only set when AI_DIGEST_ENABLED && rule.aiDigestEnabled !== false && synthesisLevel !== 3. The persisted envelope always carries a digest.lead (the L3 stub included), so reading from the briefByUser entry — NOT the envelope — is what keeps L3/opt-out users from a fake "Executive Summary". - Parity log: channels_equal is now `n/a` (not a misleading `false`) when there is no channel synthesis (L3 / opt-out / compose-miss), so those users stop tripping a false PARITY REGRESSION every tick. The remaining PARITY REGRESSION branch fires only on a genuine post-compose envelope mutation — which after this change should be unreachable for L1/L2. Legitimate non-violations preserved: generateDigestProsePublic stays (separate public/share synthesis), and the L1→L2 fallback inside runSynthesisWithFallback may still call generateDigestProse twice within one canonical attempt. Tests: 2 new enrichBriefEnvelopeWithLLM cases — skipDigestProse:true suppresses the generateDigestProse call + leaves digest untouched + still enriches whyMatters; default (no opts) still synthesises (back-compat). 93/93 brief-llm tests, 8702/8702 full unit suite, typecheck clean. Plan: docs/plans/2026-05-14-001-fix-brief-pipeline-parity-grounding-opinion-plan.md (F1, Phase 1). Codex-reviewed (3 rounds, APPROVED). Phase 1.5 (F5 root cause) finding: a harness proved orderBriefCandidates cannot produce the May 14 interleaved severity walk under any topic assignment — H2 (comparator bug) ruled out, H1 confirmed. The comparator is untouched; F5 gets a post-deploy verification step. * fix(brief): compose-miss no longer trips a false winner_match PARITY REGRESSION Review catch on PR #3685. The Phase-1 change fixed the channels_equal false-positive for L3/opt-out, but compose-miss still tripped the OTHER alarm: when briefByUser has no entry, `brief` is undefined, so the parity block computed winnerVariant='' → winnerMatch=false → fired `PARITY REGRESSION winner_match=false` every compose-miss tick. Split the parity block: when `!brief` (compose-miss), emit an informational parity line with winner_match=n/a channels_equal=n/a reason=compose-miss and skip BOTH alarms — the compose-miss is already logged separately via `[digest] compose-miss user=…`. The winner_match alarm now only fires when `brief` exists, which is the genuine bug case (canonical-rule filter bypass / chosenVariant drift). Updated the stale "unreachable in practice" comment to note it is reached only with a present `brief`. Pre-existing bug (compose-miss always set winnerVariant=''), but this PR owns the parity-log block and Phase 1 step 5 of the plan explicitly scopes "no compose-pass result → n/a, not a false regression" — compose-miss is the same family as L3/opt-out. typecheck clean, 143/143 brief-llm + digest-orchestration tests pass. * fix(brief): synthesis-boundary adapter — un-starve the digest prompt + restore the grounding gate PR B of the brief-pipeline plan (Phase 2, F2 + F8). Stacked on PR A (#3685). ESCALATED FINDING. The plan scoped F2 as "grounding gate inert." A direct probe proved it is far worse: the synthesis prompt receives ZERO story content. `buildDigest` pushes stories shaped { hash, title, link, severity, currentScore, mentionCount, phase, sources, description }. `digestFor` → `buildDigest` with no transform, so `runSynthesisWithFallback` gets that raw shape. But `buildDigestPrompt`, `checkLeadGrounding`, and `hashDigestInput` all read { headline, threatLevel, source, category, country } — fields the raw shape does not have. The title→headline / severity→threatLevel mapping only happens inside `digestStoryToUpstreamTopStory`, which runs at COMPOSE time, AFTER synthesis. Probe — feeding the verbatim buildDigest shape into buildDigestPrompt: 01. [h:abc12345] [] undefined — undefined · undefined · undefined 02. [h:def67890] [] undefined — undefined · undefined · undefined Every story line is "undefined". The model gets the sensitivity level, the greeting, and 12 lines of "[h:hash] [] undefined" — no story content. It confabulates the entire lead/threads/signals from training-data priors. This — not F1 alone — is the direct cause of the May 12 Biden+crypto hallucination and the May 14 "Ukraine attacks Russian energy" confabulation. checkLeadGrounding seeing an empty headline and skipping is the SAME mismatch, downstream. Fix — one synthesis-boundary adapter: - `digestStoryToSynthesisShape` (brief-compose.mjs) maps the raw buildDigest story → { headline, threatLevel, source, category, country, hash }. The headline gets the same prefix/suffix cleanup the magazine headline gets (so the lead grounds against the same text the reader sees). category/country default to General/Global (matching digestStoryToUpstreamTopStory + filterTopStories). - F8: every free-text field (headline, source, category, country) is run through `sanitizeForPrompt` — the digest-prose prompt carries the reader's profile context, so an unsanitised hostile RSS <title> was a prompt-injection vector. threatLevel (enum) and hash (digest) are not sanitised. - Wired at the single synthesis boundary in composeAndStoreBriefForUser: winnerStories → synthesisStories once, passed to BOTH runSynthesisWithFallback (L1 + L2-slice inherit) AND generateDigestProsePublic. composeBriefFromDigestStories keeps the raw winnerStories — digestStoryToUpstreamTopStory expects it. Tests: - 6 digestStoryToSynthesisShape unit tests (mapping, headline cleanup, source fallback, explicit category/country, F8 sanitization, missing-input safety) - 4 integration tests through the FULL live-path raw→adapted shape: buildDigestPrompt produces real content not "undefined"; checkLeadGrounding RUNS (non-empty storyTokens) and both accepts a grounded lead and rejects a fabricated one; the May 12 Biden+crypto hallucination is rejected; a hostile <title> is sanitized before the prompt. 97/97 brief-llm, 76/76 brief-from-digest-stories, 8712/8712 full unit suite, typecheck clean, biome clean (2 pre-existing complexity warnings unchanged). Note: cache key prefix not bumped — the synthesis cache material (headline/threatLevel/category/country/source) goes from all-empty to real values, so old v5 rows naturally miss; one tick of re-synthesis on rollout, same cost as any shape fix. * fix(brief): address PR #3688 review — headline sanitizer + empty-source guard Greptile P2 review of digestStoryToSynthesisShape: 1. Headline sanitizer. The adapter ran the headline through the full `sanitizeForPrompt`, which strips SEMANTIC injection phrases — it would mangle a legitimate headline whose subject IS such a phrase ("Senator urges Trump to ignore all previous instructions on tariffs" → "Senator urges Trump to on tariffs"), silently degrading the synthesis this PR restores. The codebase already has `sanitizeHeadline` (structural-delimiters-only) for exactly this. Switched the headline field to it. Structural-only sanitisation left one gap: a multi-line hostile `<title>` could still break the per-story prompt line into a line-start "assistant:" role turn. Closed by normalising the title to a single line up front (a headline is one line by definition). Net: structural injection (newline role-turn, model-delimiter tokens) is neutralised; a phrase quoted as a news SUBJECT survives. 2. Empty-source guard. `sources: ['']` (and whitespace-only) passed the `typeof === 'string'` guard and produced a blank `source` field — a prompt line rendered "… — General · Global · " with a trailing empty attribution. Added a `.trim().length > 0` check so it falls back to "Multiple wires". Non-headline free-text fields (source/category/country) keep the full `sanitizeForPrompt` — they are metadata, not headlines. Tests updated: the F8 headline test now asserts the corrected behaviour (semantic phrase preserved, structural injection neutralised) and the source-fallback test covers empty/whitespace entries. 418 brief-compose/llm tests pass; biome clean.
Summary
PR A of the brief-pipeline plan (
docs/plans/2026-05-14-001-fix-brief-pipeline-parity-grounding-opinion-plan.md, Codex-approved after 3 rounds). Phase 1 + Phase 1.5.The May 14 0802 brief tripped
PARITY REGRESSIONfor a real user — the email channel lead (403 chars) and the magazine envelope lead (373 chars) were different syntheses. PR #3396's canonical-brain promise had regressed because there were three synthesis call sites, not one:runSynthesisWithFallback(ctx-aware), spliced into the envelope. Its output never survived.enrichBriefEnvelopeWithLLM— a 4-arg, NO-ctxgenerateDigestProsecall that overwroteenvelope.data.digest. The comment claimed it "only fills per-story rationales" — it lied.runSynthesisWithFallbackagainst a differentstoriespool, for the channel bodies.Compose built a ctx-aware synthesis, enrich threw it away and re-rolled context-free, send rolled a third time. At temperature 0.4 the three rolls diverge.
Fix — collapse to ONE synthesis, produced by the compose pass
enrichBriefEnvelopeWithLLMgainsopts.skipDigestProse. When set, it does ONLY per-storywhyMatters/descriptionenrichment and leavesenvelope.data.digestuntouched. The compose path passesskipDigestProse: true. Call site 2 gone.composeAndStoreBriefForUserreturns thesynthesisobject on itsbriefByUserentry (alongside the existingsynthesisLevel).runSynthesisWithFallback. It readsbriefSynthesis/briefLead/synthesisLevelfrom the compose-pass result. Call site 3 gone.briefSynthesis/briefLeadonly set whenAI_DIGEST_ENABLED && rule.aiDigestEnabled !== false && synthesisLevel !== 3. The persisted envelope always carries adigest.lead(the L3 stub included), so reading from thebriefByUserentry — not the envelope — is what keeps L3/opt-out users from a fake "Executive Summary".channels_equalis nown/a(not a misleadingfalse) when there is no channel synthesis (L3 / opt-out / compose-miss). Stops those users tripping a falsePARITY REGRESSIONevery tick. The remainingPARITY REGRESSIONbranch fires only on a genuine post-compose envelope mutation.Legitimate non-violations preserved:
generateDigestProsePublicstays (separate public/share synthesis); the L1→L2 fallback insiderunSynthesisWithFallbackmay still callgenerateDigestProsetwice within one canonical attempt.Phase 1.5 — F5 root-cause finding (investigation, no code)
A harness fed the verbatim May 14 severity walk (
CCHHHHCCCCHH) throughorderBriefCandidatesunder four topic-assignment scenarios. None reproduce the shipped interleave —bestSeverityRankis the comparator's primary block-sort key, so a high-only block can never precede a critical block. H2 (comparator bug) is ruled out; H1 confirmed. The comparator is untouched. F5 gets a post-PR-A verification step (re-fetch a freshly-composed brief; if still interleaved, probe thethreatLevelfield as seen byfilterTopStories). Full finding recorded in the plan's Phase 1.5 section.Test plan
enrichBriefEnvelopeWithLLMtests:skipDigestProse: truesuppresses thegenerateDigestProsecall + leavesdigestuntouched (same reference) + still enricheswhyMatters; default (no opts) still synthesises (back-compat)node --test tests/brief-llm.test.mjs→ 93/93 passtsx --test tests/*.test.mjs tests/*.test.mts→ 8702/8702 passnpx tsc --noEmitcleanbiome check— 2 pre-existing complexity warnings onbuildDigest/main(4-5x over limit on origin/main already; this change neither introduced nor worsened them)PARITY REGRESSIONin the cron log;channels_equal=truefor L1/L2 users,channels_equal=n/afor L3/opt-out; email lead == magazine lead byte-identicalorderBriefCandidates-clean (criticals-first within topic blocks)Not in this PR
PR B (grounding gate + prompt sanitization) and PR C (opinion exclusion + lead/card coherence) — both depend on PR A and are tracked in the plan. F5's post-deploy verification and any residual probe ride on the plan, not this PR.