fix(brief): forbid weak temporal stitching in digest lead#3769
Conversation
May 17 brief shipped a lead that stapled the WHO Ebola declaration
and the Israel/Hezbollah escalation into a single sentence with
"This declaration comes as…" — two unrelated top stories joined by
an editorial connective, with no shared actor, causal link, or
geographic theatre.
DIGEST_PROSE_SYSTEM_BASE now:
- Tells the model to lead with ONE primary story and only reference
a second when there is a substantive link (shared actor, causal
connection, direct policy consequence, same geographic theatre).
- Carries a dedicated BANNED stitching-phrases section enumerating
the common offenders ("this comes as", "meanwhile", "in other
news", "elsewhere", etc.).
Cache prefix bumped v7 → v8 (brief-llm.mjs:23 comment, :962 key)
to invalidate any v7 rows that were generated against the stitching-
tolerant prompt — without the bump, stale stitched leads would
serve for the full 4h TTL after deploy.
Tests:
- New `forbids weak stitching connectives in the lead` test asserts
the prompt contains the anti-stitching instruction, the BANNED
section, the enumerated phrases, and the substantive-link
allowlist.
- Bumped cache-prefix assertion v7 → v8 with an added negative
assertion that no v7 rows are written.
Pattern reference: PR #3751 followed the same prompt-change → cache-
prefix-bump → assertion-update pattern when category persistence
landed (v6 → v7).
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR addresses a prompt-quality regression where the May 17 brief joined two unrelated top stories (WHO Ebola declaration and Israel/Hezbollah escalation) with the stitching phrase "This declaration comes as…". The fix adds explicit anti-stitching guidance to
Confidence Score: 4/5Safe to merge — the only change to production code is the prompt text and cache key bump, both of which are self-consistent and low-risk. The prompt change and cache invalidation are tightly scoped and correct. The one gap is that the new regression test's banned-phrase loop only covers 4 of the 9 enumerated connectives, meaning a future edit that drops one of the untested phrases would pass CI unnoticed. The production change itself is solid; the test coverage is incomplete. tests/brief-llm.test.mjs — the banned-phrase assertion loop should be extended to cover all nine connectives listed in the BANNED section. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[generateDigestProse called] --> B{Cache hit?\nbrief:llm:digest:v8:...}
B -- hit --> C[Re-run shape+grounding validator]
C -- valid --> D[Return cached result]
C -- invalid --> E[Call LLM with DIGEST_PROSE_SYSTEM_BASE]
B -- miss --> E
E --> F{LLM output\nvalid & grounded?}
F -- no --> G[Return null\nno cache write]
F -- yes --> H{Lead contains\nbanned stitching phrase?}
H -- yes --> I[Model non-compliance\nspot-check after deploy]
H -- no --> J[Write v8 cache row\n4h TTL]
J --> K[Return parsed prose]
Reviews (1): Last reviewed commit: "fix(brief): forbid weak temporal stitchi..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| for (const phrase of [ | ||
| 'this comes as', | ||
| 'meanwhile', | ||
| 'in other news', | ||
| 'elsewhere', | ||
| ]) { | ||
| assert.ok( | ||
| system.toLowerCase().includes(`"${phrase}"`), | ||
| `banned-stitching section must list "${phrase}"`, | ||
| ); | ||
| } |
There was a problem hiding this comment.
The test loop only exercises 4 of the 9 phrases enumerated in the BANNED stitching phrases section — "at the same time", "across the world", "on another front", and "in a separate development" are all absent. If any of those phrases were accidentally removed from DIGEST_PROSE_SYSTEM_BASE in a future edit, the test would still pass without surfacing the regression. The PR description explicitly calls out all nine as intentionally added, so they deserve corresponding assertions.
PR #3769 review noted the regression test only checked 4 of the 10 banned phrases and scanned the entire system prompt — so a phrase removed from the dedicated BANNED section could still pass on a duplicate mention in the lead-field instructions. Fix: - Extract the BANNED stitching phrases section as a bounded substring (from the section header up to the next `Threads:` instruction bullet). - Assert all 10 banned phrases appear inside that substring, not the broader system prompt. A regression that removes a phrase from the section now fires even if the lead instruction still mentions it. 102/102 brief-llm tests pass.
1 participant
Summary
DIGEST_PROSE_SYSTEM_BASEnow explicitly forbids weak temporal connectives ("This comes as", "Meanwhile", "At the same time", "In other news", "Elsewhere", "Across the world", "On another front", "In a separate development") and tells the model to lead with one primary story when two top stories aren't substantively linked (shared actor / causal connection / direct policy consequence / same geographic theatre).scripts/lib/brief-llm.mjsso stale v7 rows generated against the stitching-tolerant prompt do not serve for the 4h TTL after deploy.Why prompt-only (not a synthesis-step rewrite)
The conflation was a stylistic failure of the existing instructions, not a missing pipeline stage. Adding a focused "no-stitching" instruction + cache bump is the cheapest fix that addresses the regression without growing prompt surface area or adding a second LLM round-trip. Pattern reference: PR #3751 followed the same prompt-change → cache-prefix-bump → assertion-update shape when category persistence landed (v6 → v7).
Files changed
scripts/lib/brief-llm.mjs—DIGEST_PROSE_SYSTEM_BASEupdated lead instruction + new BANNED stitching section; cache key v7 → v8 (line 962) + matching comment (line 23) + v8 rationale in the version-history block.tests/brief-llm.test.mjs— newforbids weak stitching connectives in the lead (anti-conflation, v8)test asserting the prompt contains the instruction text, the BANNED section, the enumerated phrases, and the substantive-link allowlist; cache-prefix assertion bumped v7 → v8 with an added negative assertion that no v7 rows are written.Test plan
node --test tests/brief-llm.test.mjs— 102/102 pass, including the new anti-stitching assertion and the v8 prefix-bump assertion.tests/brief-*.test.mjssweep — 664/670 pass; the 6 failures are pre-existingapi/brief/*.tsESM module-resolution issues unrelated to this change.