Skip to content

fix(brief): single canonical synthesis — kill the compose/enrich/send parity regression#3685

Merged
koala73 merged 2 commits into
mainfrom
fix/brief-parity-regression
May 14, 2026
Merged

fix(brief): single canonical synthesis — kill the compose/enrich/send parity regression#3685
koala73 merged 2 commits into
mainfrom
fix/brief-parity-regression

Conversation

@koala73

@koala73 koala73 commented May 14, 2026

Copy link
Copy Markdown
Owner

Summary

PR A of the brief-pipeline plan (docs/plans/2026-05-14-001-fix-brief-pipeline-parity-grounding-opinion-plan.md, Codex-approved after 3 rounds). Phase 1 + Phase 1.5.

The May 14 0802 brief tripped PARITY REGRESSION for a real user — the email channel lead (403 chars) and the magazine envelope lead (373 chars) were different syntheses. PR #3396's canonical-brain promise had regressed because there were three synthesis call sites, not one:

  1. Compose passrunSynthesisWithFallback (ctx-aware), spliced into the envelope. Its output never survived.
  2. enrichBriefEnvelopeWithLLM — a 4-arg, NO-ctx generateDigestProse call that overwrote envelope.data.digest. The comment claimed it "only fills per-story rationales" — it lied.
  3. Send passrunSynthesisWithFallback against a different stories pool, for the channel bodies.

Compose built a ctx-aware synthesis, enrich threw it away and re-rolled context-free, send rolled a third time. At temperature 0.4 the three rolls diverge.

Fix — collapse to ONE synthesis, produced by the compose pass

  • enrichBriefEnvelopeWithLLM gains opts.skipDigestProse. When set, it does ONLY per-story whyMatters/description enrichment and leaves envelope.data.digest untouched. The compose path passes skipDigestProse: true. Call site 2 gone.
  • composeAndStoreBriefForUser returns the synthesis object on its briefByUser entry (alongside the existing synthesisLevel).
  • The send pass no longer calls runSynthesisWithFallback. It reads briefSynthesis / briefLead / synthesisLevel from the compose-pass result. Call site 3 gone.
  • L3 / opt-out gate preservedbriefSynthesis/briefLead only set when AI_DIGEST_ENABLED && rule.aiDigestEnabled !== false && synthesisLevel !== 3. The persisted envelope always carries a digest.lead (the L3 stub included), so reading from the briefByUser entry — not the envelope — is what keeps L3/opt-out users from a fake "Executive Summary".
  • Parity log: channels_equal is now n/a (not a misleading false) when there is no channel synthesis (L3 / opt-out / compose-miss). Stops those users tripping a false PARITY REGRESSION every tick. The remaining PARITY REGRESSION branch fires only on a genuine post-compose envelope mutation.

Legitimate non-violations preserved: generateDigestProsePublic stays (separate public/share synthesis); the L1→L2 fallback inside runSynthesisWithFallback may still call generateDigestProse twice within one canonical attempt.

Phase 1.5 — F5 root-cause finding (investigation, no code)

A harness fed the verbatim May 14 severity walk (CCHHHHCCCCHH) through orderBriefCandidates under four topic-assignment scenarios. None reproduce the shipped interleavebestSeverityRank is the comparator's primary block-sort key, so a high-only block can never precede a critical block. H2 (comparator bug) is ruled out; H1 confirmed. The comparator is untouched. F5 gets a post-PR-A verification step (re-fetch a freshly-composed brief; if still interleaved, probe the threatLevel field as seen by filterTopStories). Full finding recorded in the plan's Phase 1.5 section.

Test plan

  • 2 new enrichBriefEnvelopeWithLLM tests: skipDigestProse: true suppresses the generateDigestProse call + leaves digest untouched (same reference) + still enriches whyMatters; default (no opts) still synthesises (back-compat)
  • node --test tests/brief-llm.test.mjs → 93/93 pass
  • tsx --test tests/*.test.mjs tests/*.test.mts → 8702/8702 pass
  • npx tsc --noEmit clean
  • biome check — 2 pre-existing complexity warnings on buildDigest/main (4-5x over limit on origin/main already; this change neither introduced nor worsened them)
  • After deploy — verify on a freshly-composed brief: zero PARITY REGRESSION in the cron log; channels_equal=true for L1/L2 users, channels_equal=n/a for L3/opt-out; email lead == magazine lead byte-identical
  • After deploy — F5 verification: re-fetch a brief, confirm severity walk is orderBriefCandidates-clean (criticals-first within topic blocks)

Not in this PR

PR B (grounding gate + prompt sanitization) and PR C (opinion exclusion + lead/card coherence) — both depend on PR A and are tracked in the plan. F5's post-deploy verification and any residual probe ride on the plan, not this PR.

… parity regression

The May 14 0802 brief tripped `PARITY REGRESSION` for a real user:
the email channel lead (403 chars) and the magazine envelope lead
(373 chars) were different syntheses. The canonical-brain promise
from PR #3396 ("one synthesis, every channel byte-identical") had
regressed — there were THREE synthesis call sites, not one:

  1. Compose pass — runSynthesisWithFallback (seed-digest:1530),
     ctx-aware, spliced into the envelope. Its output never survived.
  2. enrichBriefEnvelopeWithLLM — generateDigestProse (brief-llm.mjs)
     with a 4-arg NO-ctx call that OVERWROTE envelope.data.digest.
     The comment claimed "this pass only fills per-story rationales";
     it lied — it re-synthesised the whole digest prose.
  3. Send pass — runSynthesisWithFallback (seed-digest:1949) against
     a different `stories` pool, for the channel bodies.

So compose built a ctx-aware synthesis, enrich threw it away and
re-rolled context-free, and send rolled a third time. At temperature
0.4 the three rolls diverge.

Fix — collapse to ONE synthesis, produced by the compose pass:

- enrichBriefEnvelopeWithLLM gains `opts.skipDigestProse`. When set,
  it does ONLY per-story whyMatters/description enrichment and leaves
  envelope.data.digest untouched — no second generateDigestProse
  call. The compose path passes skipDigestProse:true (the canonical
  synthesis is already spliced in). Call site 2 is gone.

- composeAndStoreBriefForUser now returns the `synthesis` object on
  its briefByUser entry (alongside the existing synthesisLevel).

- The send pass no longer calls runSynthesisWithFallback. It reads
  briefSynthesis / briefLead / synthesisLevel from the compose-pass
  result on the briefByUser entry. Call site 3 is gone.

- L3 / opt-out gate preserved: briefSynthesis/briefLead are only set
  when AI_DIGEST_ENABLED && rule.aiDigestEnabled !== false &&
  synthesisLevel !== 3. The persisted envelope always carries a
  digest.lead (the L3 stub included), so reading from the briefByUser
  entry — NOT the envelope — is what keeps L3/opt-out users from a
  fake "Executive Summary".

- Parity log: channels_equal is now `n/a` (not a misleading `false`)
  when there is no channel synthesis (L3 / opt-out / compose-miss),
  so those users stop tripping a false PARITY REGRESSION every tick.
  The remaining PARITY REGRESSION branch fires only on a genuine
  post-compose envelope mutation — which after this change should be
  unreachable for L1/L2.

Legitimate non-violations preserved: generateDigestProsePublic stays
(separate public/share synthesis), and the L1→L2 fallback inside
runSynthesisWithFallback may still call generateDigestProse twice
within one canonical attempt.

Tests: 2 new enrichBriefEnvelopeWithLLM cases — skipDigestProse:true
suppresses the generateDigestProse call + leaves digest untouched +
still enriches whyMatters; default (no opts) still synthesises
(back-compat). 93/93 brief-llm tests, 8702/8702 full unit suite,
typecheck clean.

Plan: docs/plans/2026-05-14-001-fix-brief-pipeline-parity-grounding-opinion-plan.md
(F1, Phase 1). Codex-reviewed (3 rounds, APPROVED).

Phase 1.5 (F5 root cause) finding: a harness proved orderBriefCandidates
cannot produce the May 14 interleaved severity walk under any topic
assignment — H2 (comparator bug) ruled out, H1 confirmed. The
comparator is untouched; F5 gets a post-deploy verification step.
@vercel

vercel Bot commented May 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview May 14, 2026 6:22am

Request Review

@greptile-apps

greptile-apps Bot commented May 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR collapses three independent synthesis call sites down to one canonical synthesis produced in the compose pass, eliminating the compose↔send parity regression that caused the email and magazine leads to diverge at temperature 0.4. It also fixes a false PARITY REGRESSION alarm that fired every tick for L3/opt-out users.

  • enrichBriefEnvelopeWithLLM gains opts.skipDigestProse — when true, the function skips the digest-prose LLM call entirely and passes envelope.data.digest through untouched, preventing "call site 2" from overwriting the compose-pass synthesis with a context-free re-roll.
  • composeAndStoreBriefForUser now returns the synthesis object on its briefByUser entry; the send pass reads brief?.synthesis directly instead of calling runSynthesisWithFallback again, eliminating "call site 3".
  • Parity log introduces hasChannelSynthesis to gate channelsEqual as 'n/a' when no channel synthesis exists (L3 stub, aiDigest opt-out, or compose-miss), stopping false PARITY REGRESSION alarms for those users; the alarm now fires only for a genuine post-compose envelope mutation.

Confidence Score: 4/5

Safe to merge — the core parity fix is mechanically simple (pass synthesis through an in-memory Map instead of re-rolling it), tests are clean, and the trade-off for compose-miss users is explicitly documented and self-healing.

The three-to-one synthesis consolidation is correct: the skipDigestProse flag prevents the compose-pass synthesis from being silently overwritten, and the send pass reading brief?.synthesis directly is simpler and more reliable than re-calling runSynthesisWithFallback. The channelsEqual === false guard is stricter than the old !channelsEqual && briefLead && envLead — it will now catch a mismatch even when envLead is an empty string, which is more correct but a subtle surface-area change worth watching post-deploy.

The send-pass block in scripts/seed-digest-notifications.mjs around the briefSynthesis/briefLead assignment and the parity-log conditional — specifically the compose-miss fallback path where brief is undefined and synthesisLevel defaults to 3.

Important Files Changed

Filename Overview
scripts/lib/brief-llm.mjs Adds opts.skipDigestProse parameter to enrichBriefEnvelopeWithLLM; implementation is correct — digest is passed through by reference when the flag is set, and per-story enrichment still runs.
scripts/seed-digest-notifications.mjs Compose pass now returns synthesis on the briefByUser entry; send pass reads it directly instead of re-running runSynthesisWithFallback; parity log guards channelsEqual correctly. Compose-miss users intentionally get no editorial block this tick — trade-off is documented.
tests/brief-llm.test.mjs Two new tests cover skipDigestProse: true (no prose LLM call, digest reference preserved, per-story enrichment still runs) and the default path (digest prose still synthesised); both are well-structured and validate the key invariants.

Sequence Diagram

sequenceDiagram
    participant CP as Compose Pass
    participant BU as briefByUser (in-memory)
    participant EN as enrichBriefEnvelopeWithLLM
    participant SP as Send Pass

    CP->>CP: runSynthesisWithFallback(userId, winnerStories, ctx)
    CP->>EN: enrichBriefEnvelopeWithLLM(envelope, rule, deps, skipDigestProse: true)
    Note over EN: Skips generateDigestProse
    EN-->>CP: enriched envelope (digest untouched)
    CP->>BU: "briefByUser.set(userId, {envelope, synthesis, synthesisLevel})"

    SP->>BU: "brief = briefByUser.get(userId)"
    alt L1/L2 user
        SP->>SP: "briefSynthesis = brief.synthesis"
        Note over SP: channelsEqual = briefLead === envLead
    else L3 / opt-out / compose-miss
        SP->>SP: "briefLead = null"
        Note over SP: channelsEqual = 'n/a'
    end
    SP->>SP: Render channel bodies using briefSynthesis
Loading

Reviews (1): Last reviewed commit: "fix(brief): single canonical synthesis —..." | Re-trigger Greptile

…REGRESSION

Review catch on PR #3685. The Phase-1 change fixed the channels_equal
false-positive for L3/opt-out, but compose-miss still tripped the
OTHER alarm: when briefByUser has no entry, `brief` is undefined, so
the parity block computed winnerVariant='' → winnerMatch=false →
fired `PARITY REGRESSION winner_match=false` every compose-miss tick.

Split the parity block: when `!brief` (compose-miss), emit an
informational parity line with winner_match=n/a channels_equal=n/a
reason=compose-miss and skip BOTH alarms — the compose-miss is
already logged separately via `[digest] compose-miss user=…`. The
winner_match alarm now only fires when `brief` exists, which is the
genuine bug case (canonical-rule filter bypass / chosenVariant
drift). Updated the stale "unreachable in practice" comment to note
it is reached only with a present `brief`.

Pre-existing bug (compose-miss always set winnerVariant=''), but
this PR owns the parity-log block and Phase 1 step 5 of the plan
explicitly scopes "no compose-pass result → n/a, not a false
regression" — compose-miss is the same family as L3/opt-out.

typecheck clean, 143/143 brief-llm + digest-orchestration tests pass.
@koala73
koala73 merged commit a5da3a2 into main May 14, 2026
11 checks passed
@koala73
koala73 deleted the fix/brief-parity-regression branch May 14, 2026 14:04
koala73 added a commit that referenced this pull request May 14, 2026
…+ restore the grounding gate (#3688)

* fix(brief): single canonical synthesis — kill the compose/enrich/send parity regression

The May 14 0802 brief tripped `PARITY REGRESSION` for a real user:
the email channel lead (403 chars) and the magazine envelope lead
(373 chars) were different syntheses. The canonical-brain promise
from PR #3396 ("one synthesis, every channel byte-identical") had
regressed — there were THREE synthesis call sites, not one:

  1. Compose pass — runSynthesisWithFallback (seed-digest:1530),
     ctx-aware, spliced into the envelope. Its output never survived.
  2. enrichBriefEnvelopeWithLLM — generateDigestProse (brief-llm.mjs)
     with a 4-arg NO-ctx call that OVERWROTE envelope.data.digest.
     The comment claimed "this pass only fills per-story rationales";
     it lied — it re-synthesised the whole digest prose.
  3. Send pass — runSynthesisWithFallback (seed-digest:1949) against
     a different `stories` pool, for the channel bodies.

So compose built a ctx-aware synthesis, enrich threw it away and
re-rolled context-free, and send rolled a third time. At temperature
0.4 the three rolls diverge.

Fix — collapse to ONE synthesis, produced by the compose pass:

- enrichBriefEnvelopeWithLLM gains `opts.skipDigestProse`. When set,
  it does ONLY per-story whyMatters/description enrichment and leaves
  envelope.data.digest untouched — no second generateDigestProse
  call. The compose path passes skipDigestProse:true (the canonical
  synthesis is already spliced in). Call site 2 is gone.

- composeAndStoreBriefForUser now returns the `synthesis` object on
  its briefByUser entry (alongside the existing synthesisLevel).

- The send pass no longer calls runSynthesisWithFallback. It reads
  briefSynthesis / briefLead / synthesisLevel from the compose-pass
  result on the briefByUser entry. Call site 3 is gone.

- L3 / opt-out gate preserved: briefSynthesis/briefLead are only set
  when AI_DIGEST_ENABLED && rule.aiDigestEnabled !== false &&
  synthesisLevel !== 3. The persisted envelope always carries a
  digest.lead (the L3 stub included), so reading from the briefByUser
  entry — NOT the envelope — is what keeps L3/opt-out users from a
  fake "Executive Summary".

- Parity log: channels_equal is now `n/a` (not a misleading `false`)
  when there is no channel synthesis (L3 / opt-out / compose-miss),
  so those users stop tripping a false PARITY REGRESSION every tick.
  The remaining PARITY REGRESSION branch fires only on a genuine
  post-compose envelope mutation — which after this change should be
  unreachable for L1/L2.

Legitimate non-violations preserved: generateDigestProsePublic stays
(separate public/share synthesis), and the L1→L2 fallback inside
runSynthesisWithFallback may still call generateDigestProse twice
within one canonical attempt.

Tests: 2 new enrichBriefEnvelopeWithLLM cases — skipDigestProse:true
suppresses the generateDigestProse call + leaves digest untouched +
still enriches whyMatters; default (no opts) still synthesises
(back-compat). 93/93 brief-llm tests, 8702/8702 full unit suite,
typecheck clean.

Plan: docs/plans/2026-05-14-001-fix-brief-pipeline-parity-grounding-opinion-plan.md
(F1, Phase 1). Codex-reviewed (3 rounds, APPROVED).

Phase 1.5 (F5 root cause) finding: a harness proved orderBriefCandidates
cannot produce the May 14 interleaved severity walk under any topic
assignment — H2 (comparator bug) ruled out, H1 confirmed. The
comparator is untouched; F5 gets a post-deploy verification step.

* fix(brief): compose-miss no longer trips a false winner_match PARITY REGRESSION

Review catch on PR #3685. The Phase-1 change fixed the channels_equal
false-positive for L3/opt-out, but compose-miss still tripped the
OTHER alarm: when briefByUser has no entry, `brief` is undefined, so
the parity block computed winnerVariant='' → winnerMatch=false →
fired `PARITY REGRESSION winner_match=false` every compose-miss tick.

Split the parity block: when `!brief` (compose-miss), emit an
informational parity line with winner_match=n/a channels_equal=n/a
reason=compose-miss and skip BOTH alarms — the compose-miss is
already logged separately via `[digest] compose-miss user=…`. The
winner_match alarm now only fires when `brief` exists, which is the
genuine bug case (canonical-rule filter bypass / chosenVariant
drift). Updated the stale "unreachable in practice" comment to note
it is reached only with a present `brief`.

Pre-existing bug (compose-miss always set winnerVariant=''), but
this PR owns the parity-log block and Phase 1 step 5 of the plan
explicitly scopes "no compose-pass result → n/a, not a false
regression" — compose-miss is the same family as L3/opt-out.

typecheck clean, 143/143 brief-llm + digest-orchestration tests pass.

* fix(brief): synthesis-boundary adapter — un-starve the digest prompt + restore the grounding gate

PR B of the brief-pipeline plan (Phase 2, F2 + F8). Stacked on PR A
(#3685).

ESCALATED FINDING. The plan scoped F2 as "grounding gate inert."
A direct probe proved it is far worse: the synthesis prompt
receives ZERO story content.

`buildDigest` pushes stories shaped { hash, title, link, severity,
currentScore, mentionCount, phase, sources, description }. `digestFor`
→ `buildDigest` with no transform, so `runSynthesisWithFallback`
gets that raw shape. But `buildDigestPrompt`, `checkLeadGrounding`,
and `hashDigestInput` all read { headline, threatLevel, source,
category, country } — fields the raw shape does not have. The
title→headline / severity→threatLevel mapping only happens inside
`digestStoryToUpstreamTopStory`, which runs at COMPOSE time, AFTER
synthesis.

Probe — feeding the verbatim buildDigest shape into buildDigestPrompt:

  01. [h:abc12345] [] undefined — undefined · undefined · undefined
  02. [h:def67890] [] undefined — undefined · undefined · undefined

Every story line is "undefined". The model gets the sensitivity
level, the greeting, and 12 lines of "[h:hash] [] undefined" — no
story content. It confabulates the entire lead/threads/signals from
training-data priors. This — not F1 alone — is the direct cause of
the May 12 Biden+crypto hallucination and the May 14 "Ukraine
attacks Russian energy" confabulation. checkLeadGrounding seeing an
empty headline and skipping is the SAME mismatch, downstream.

Fix — one synthesis-boundary adapter:

- `digestStoryToSynthesisShape` (brief-compose.mjs) maps the raw
  buildDigest story → { headline, threatLevel, source, category,
  country, hash }. The headline gets the same prefix/suffix cleanup
  the magazine headline gets (so the lead grounds against the same
  text the reader sees). category/country default to General/Global
  (matching digestStoryToUpstreamTopStory + filterTopStories).

- F8: every free-text field (headline, source, category, country) is
  run through `sanitizeForPrompt` — the digest-prose prompt carries
  the reader's profile context, so an unsanitised hostile RSS <title>
  was a prompt-injection vector. threatLevel (enum) and hash (digest)
  are not sanitised.

- Wired at the single synthesis boundary in composeAndStoreBriefForUser:
  winnerStories → synthesisStories once, passed to BOTH
  runSynthesisWithFallback (L1 + L2-slice inherit) AND
  generateDigestProsePublic. composeBriefFromDigestStories keeps the
  raw winnerStories — digestStoryToUpstreamTopStory expects it.

Tests:
- 6 digestStoryToSynthesisShape unit tests (mapping, headline
  cleanup, source fallback, explicit category/country, F8
  sanitization, missing-input safety)
- 4 integration tests through the FULL live-path raw→adapted shape:
  buildDigestPrompt produces real content not "undefined";
  checkLeadGrounding RUNS (non-empty storyTokens) and both accepts a
  grounded lead and rejects a fabricated one; the May 12 Biden+crypto
  hallucination is rejected; a hostile <title> is sanitized before
  the prompt.

97/97 brief-llm, 76/76 brief-from-digest-stories, 8712/8712 full
unit suite, typecheck clean, biome clean (2 pre-existing complexity
warnings unchanged).

Note: cache key prefix not bumped — the synthesis cache material
(headline/threatLevel/category/country/source) goes from all-empty
to real values, so old v5 rows naturally miss; one tick of
re-synthesis on rollout, same cost as any shape fix.

* fix(brief): address PR #3688 review — headline sanitizer + empty-source guard

Greptile P2 review of digestStoryToSynthesisShape:

1. Headline sanitizer. The adapter ran the headline through the full
   `sanitizeForPrompt`, which strips SEMANTIC injection phrases — it
   would mangle a legitimate headline whose subject IS such a phrase
   ("Senator urges Trump to ignore all previous instructions on
   tariffs" → "Senator urges Trump to on tariffs"), silently degrading
   the synthesis this PR restores. The codebase already has
   `sanitizeHeadline` (structural-delimiters-only) for exactly this.
   Switched the headline field to it.

   Structural-only sanitisation left one gap: a multi-line hostile
   `<title>` could still break the per-story prompt line into a
   line-start "assistant:" role turn. Closed by normalising the title
   to a single line up front (a headline is one line by definition).
   Net: structural injection (newline role-turn, model-delimiter
   tokens) is neutralised; a phrase quoted as a news SUBJECT survives.

2. Empty-source guard. `sources: ['']` (and whitespace-only) passed
   the `typeof === 'string'` guard and produced a blank `source`
   field — a prompt line rendered "… — General · Global · " with a
   trailing empty attribution. Added a `.trim().length > 0` check so
   it falls back to "Multiple wires".

Non-headline free-text fields (source/category/country) keep the full
`sanitizeForPrompt` — they are metadata, not headlines. Tests updated:
the F8 headline test now asserts the corrected behaviour (semantic
phrase preserved, structural injection neutralised) and the
source-fallback test covers empty/whitespace entries.

418 brief-compose/llm tests pass; biome clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant