…lead divergence (koala73#3396)

* feat(brief-llm): canonical synthesis prompt + v3 cache key

Extends generateDigestProse to be the single source of truth for
brief executive-summary synthesis (canonicalises what was previously
split between brief-llm's generateDigestProse and seed-digest-
notifications.mjs's generateAISummary). Ports Brain B's prompt
features into buildDigestPrompt:

- ctx={profile, greeting, isPublic} parameter (back-compat: 4-arg
  callers behave like today)
- per-story severity uppercased + short-hash prefix [h:XXXX] so the
  model can emit rankedStoryHashes for stable re-ranking
- profile lines + greeting opener appear only when ctx.isPublic !== true

validateDigestProseShape gains optional rankedStoryHashes (≥4-char
strings, capped to MAX_STORIES_PER_USER × 2). v2-shaped rows still
pass — field defaults to [].

hashDigestInput v3:
- material includes profile-SHA, greeting bucket, isPublic flag,
  per-story hash
- isPublic=true substitutes literal 'public' for userId in the cache
  key so all share-URL readers of the same (date, sensitivity, pool)
  hit ONE cache row (no PII in public cache key)

Adds generateDigestProsePublic(stories, sensitivity, deps) wrapper —
no userId param by design — for the share-URL surface.

Cache prefix bumped brief:llm:digest:v2 → v3. v2 rows expire on TTL.
Per the v1→v2 precedent (see hashDigestInput comment), one-tick cost
on rollout is acceptable for cache-key correctness.

Tests: 72/72 passing in tests/brief-llm.test.mjs (8 new for the v3
behaviors), full data suite 6952/6952.

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 1, Codex-approved (5 rounds).

* feat(brief): envelope v3 — adds digest.publicLead for share-URL surface

Bumps BRIEF_ENVELOPE_VERSION 2 → 3. Adds optional
BriefDigest.publicLead — non-personalised executive lead generated
by generateDigestProsePublic (already in this branch from the
previous commit) for the public share-URL surface. Personalised
`lead` is the canonical synthesis for authenticated channels;
publicLead is its profile-stripped sibling so api/brief/public/*
never serves user-specific content (watched assets/regions).

SUPPORTED_ENVELOPE_VERSIONS = [1, 2, 3] keeps v1 + v2 envelopes
in the 7-day TTL window readable through the rollout — the
composer only ever writes the current version, but readers must
tolerate older shapes that haven't expired yet. Same rollout
pattern used at the v1 → v2 bump.

Renderer changes (server/_shared/brief-render.js):
- ALLOWED_DIGEST_KEYS gains 'publicLead' (closed-key-set still
  enforced; v2 envelopes pass because publicLead === undefined is
  the v2 shape).
- assertBriefEnvelope: new isNonEmptyString check on publicLead
  when present. Type contract enforced; absence is OK.

Tests (tests/brief-magazine-render.test.mjs):
- New describe block "v3 publicLead field": v3 envelope renders;
  malformed publicLead rejected; v2 envelope still passes; ad-hoc
  digest keys (e.g. synthesisLevel) still rejected — confirming
  the closed-key-set defense holds for the cron-local-only fields
  the orchestrator must NOT persist.
- BRIEF_ENVELOPE_VERSION pin updated 2 → 3 with rollout-rationale
  comment.

Test results: 182 brief-related tests pass; full data suite
6956/6956.

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 2, Codex Round-3 Medium #2.

* feat(brief): synthesis splice + rankedStoryHashes pre-cap re-order

Plumbs the canonical synthesis output (lead, threads, signals,
publicLead, rankedStoryHashes from generateDigestProse) through the
pure composer so the orchestration layer can hand pre-resolved data
into envelope.digest. Composer stays sync / no I/O — Codex Round-2
High #2 honored.

Changes:

scripts/lib/brief-compose.mjs:
- digestStoryToUpstreamTopStory now emits `hash` (the digest story's
  stable identifier, falls back to titleHash when absent). Without
  this, rankedStoryHashes from the LLM has nothing to match against.
- composeBriefFromDigestStories accepts opts.synthesis = {lead,
  threads, signals, rankedStoryHashes?, publicLead?}. When passed,
  splices into envelope.digest after the stub is built. Partial
  synthesis (e.g. only `lead` populated) keeps stub defaults for the
  other fields — graceful degradation when L2 fallback fires.

shared/brief-filter.js:
- filterTopStories accepts optional rankedStoryHashes. New helper
  applyRankedOrder re-orders stories by short-hash prefix match
  BEFORE the cap is applied, so the model's editorial judgment of
  importance survives MAX_STORIES_PER_USER. Stable for ties; stories
  not in the ranking come after in original order. Empty/missing
  ranking is a no-op (legacy callers unchanged).

shared/brief-filter.d.ts:
- filterTopStories signature gains rankedStoryHashes?: string[].
- UpstreamTopStory gains hash?: unknown (carried through from
  digestStoryToUpstreamTopStory).

Tests added (tests/brief-from-digest-stories.test.mjs):
- synthesis substitutes lead/threads/signals/publicLead.
- legacy 4-arg callers (no synthesis) keep stub lead.
- partial synthesis (only lead) keeps stub threads/signals.
- rankedStoryHashes re-orders pool before cap.
- short-hash prefix match (model emits 8 chars; story carries full).
- unranked stories go after in original order.

Test results: 33/33 in brief-from-digest-stories; 182/182 across all
brief tests; full data suite 6956/6956.

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 3, Codex Round-2 Low + Round-2 High #2.

* feat(brief): single canonical synthesis per user; rewire all channels

Restructures the digest cron's per-user compose + send loops to
produce ONE canonical synthesis per user per issueSlot — the lead
text every channel (email HTML, plain-text, Telegram, Slack,
Discord, webhook) and the magazine show is byte-identical. This
eliminates the "two-brain" divergence that was producing different
exec summaries on different surfaces (observed 2026-04-25 0802).

Architecture:

composeBriefsForRun (orchestration):
- Pre-annotates every eligible rule with lastSentAt + isDue once,
  before the per-user pass. Same getLastSentAt helper the send loop
  uses so compose + send agree on lastSentAt for every rule.

composeAndStoreBriefForUser (per-user):
- Two-pass winner walk: try DUE rules first (sortedDue), fall back
  to ALL eligible rules (sortedAll) for compose-only ticks.
  Preserves today's dashboard refresh contract for weekly /
  twice_daily users on non-due ticks (Codex Round-4 High #1).
- Within each pass, walk by compareRules priority and pick the
  FIRST candidate with a non-empty pool — mirrors today's behavior
  at scripts/seed-digest-notifications.mjs:1044 and prevents the
  "highest-priority but empty pool" edge case (Codex Round-4
  Medium #2).
- Three-level synthesis fallback chain:
    L1: generateDigestProse(fullPool, ctx={profile,greeting,!public})
    L2: generateDigestProse(envelope-sized slice, ctx={})
    L3: stub from assembleStubbedBriefEnvelope
  Distinct log lines per fallback level so ops can quantify
  failure-mode distribution.
- Generates publicLead in parallel via generateDigestProsePublic
  (no userId param; cache-shared across all share-URL readers).
- Splices synthesis into envelope via composer's optional
  `synthesis` arg (Step 3); rankedStoryHashes re-orders the pool
  BEFORE the cap so editorial importance survives MAX_STORIES.
- synthesisLevel stored in the cron-local briefByUser entry — NOT
  persisted in the envelope (renderer's assertNoExtraKeys would
  reject; Codex Round-2 Medium #5).

Send loop:
- Reads lastSentAt via shared getLastSentAt helper (single source
  of truth with compose flow).
- briefLead = brief?.envelope?.data?.digest?.lead — the canonical
  lead. Passed to buildChannelBodies (text/Telegram/Slack/Discord),
  injectEmailSummary (HTML email), and sendWebhook (webhook
  payload's `summary` field). All-channel parity (Codex Round-1
  Medium #6).
- Subject ternary reads cron-local synthesisLevel: 1 or 2 →
  "Intelligence Brief", 3 → "Digest" (preserves today's UX for
  fallback paths; Codex Round-1 Missing #5).

Removed:
- generateAISummary() — the second LLM call that produced the
  divergent email lead. ~85 lines.
- AI_SUMMARY_CACHE_TTL constant — no longer referenced. The
  digest:ai-summary:v1:* cache rows expire on their existing 1h
  TTL (no cleanup pass).

Helpers added:
- getLastSentAt(rule) — extracted Upstash GET for digest:last-sent
  so compose + send both call one source of truth.
- buildSynthesisCtx(rule, nowMs) — formats profile + greeting for
  the canonical synthesis call. Preserves all today's prefs-fetch
  failure-mode behavior.

Composer:
- compareRules now exported from scripts/lib/brief-compose.mjs so
  the cron can sort each pass identically to groupEligibleRulesByUser.

Test results: full data suite 6962/6962 (was 6956 pre-Step 4; +6
new compose-synthesis tests from Step 3).

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Steps 4 + 4b. Codex-approved (5 rounds).

* fix(brief-render): public-share lead fail-safe — never leak personalised lead

Public-share render path (api/brief/public/[hash].ts → renderer
publicMode=true) MUST NEVER serve the personalised digest.lead
because that string can carry profile context — watched assets,
saved-region names, etc. — written by generateDigestProse with
ctx.profile populated.

Previously: redactForPublic redacted user.name and stories.whyMatters
but passed digest.lead through unchanged. Codex Round-2 High
(security finding).

Now (v3 envelope contract):
- redactForPublic substitutes digest.lead = digest.publicLead when
  the v3 envelope carries one (generated by generateDigestProsePublic
  with profile=null, cache-shared across all public readers).
- When publicLead is absent (v2 envelope still in TTL window OR v3
  envelope where publicLead generation failed), redactForPublic sets
  digest.lead to empty string.
- renderDigestGreeting: when lead is empty, OMIT the <blockquote>
  pull-quote entirely. Page still renders complete (greeting +
  horizontal rule), just without the italic lead block.
- NEVER falls back to the original personalised lead.

assertBriefEnvelope still validates publicLead's contract (when
present, must be a non-empty string) BEFORE redactForPublic runs,
so a malformed publicLead throws before any leak risk.

Tests added (tests/brief-magazine-render.test.mjs):
- v3 envelope renders publicLead in pull-quote, personalised lead
  text never appears.
- v2 envelope (no publicLead) omits pull-quote; rest of page
  intact.
- empty-string publicLead rejected by validator (defensive).
- private render still uses personalised lead.

Test results: 68 brief-magazine-render tests pass; full data suite
remains green from prior commit.

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 5, Codex Round-2 High (security).

* feat(digest): brief lead parity log + extra acceptance tests

Adds the parity-contract observability line and supplementary
acceptance tests for the canonical synthesis path.

Parity log (per send, after successful delivery):
  [digest] brief lead parity user=<id> rule=<v>:<s>:<lang>
    synthesis_level=<1|2|3> exec_len=<n> brief_lead_len=<n>
    channels_equal=<bool> public_lead_len=<n>

When channels_equal=false an extra WARN line fires —
"PARITY REGRESSION user=… — email lead != envelope lead." Sentry's
existing console-breadcrumb hook lifts this without an explicit
captureMessage call. Plan acceptance criterion A5.

Tests added (tests/brief-llm.test.mjs, +9):
- generateDigestProsePublic: two distinct callers with identical
  (sensitivity, story-pool) hit the SAME cache row (per Codex
  Round-2 Medium #4 — "no PII in public cache key").
- public + private writes never collide on cache key (defensive).
- greeting bucket change re-keys the personalised cache (Brain B
  parity).
- profile change re-keys the personalised cache.
- v3 cache prefix used (no v2 writes).

Test results: 77/77 in brief-llm; full data suite 6971/6971
(was 6962 pre-Step-7; +9 new public-cache tests).

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Steps 6 (partial) + 7. Acceptance A5, A6.g, A6.f.

* test(digest): backfill A6.h/i/l/m acceptance tests via helper extraction

* fix(brief): close two correctness regressions on multi-rule + public surface

Two findings from human review of the canonical-synthesis PR:

1. Public-share redaction leaked personalised signals + threads.
   The new prompt explicitly personalises both `lead` and `signals`
   ("personalise lead and signals"), but redactForPublic only
   substituted `lead` — leaving `signals` and `threads` intact.
   Public renderer's hasSignals gate would emit the signals page
   whenever `digest.signals.length > 0`, exposing watched-asset /
   region phrasing to anonymous readers. Same privacy bug class
   the original PR was meant to close, just on different fields.

2. Multi-rule users got cross-pool lead/storyList mismatch.
   composeAndStoreBriefForUser picks ONE winning rule for the
   canonical envelope. The send loop then injected that ONE
   `briefLead` into every due rule's channel body — even though
   each rule's storyList came from its own (per-rule) digest pool.
   Multi-rule users (e.g. `full` + `finance`) ended up with email
   bodies leading on geopolitics while listing finance stories.
   Cross-rule editorial mismatch reintroduced after the cross-
   surface fix.

Fix 1 — public signals + threads:
- Envelope shape: BriefDigest gains `publicSignals?: string[]` +
  `publicThreads?: BriefThread[]` (sibling fields to publicLead).
  Renderer's ALLOWED_DIGEST_KEYS extended; assertBriefEnvelope
  validates them when present.
- generateDigestProsePublic already returned a full prose object
  (lead + signals + threads) — orchestration now captures all
  three instead of just `.lead`. Composer splices each into its
  envelope slot.
- redactForPublic substitutes:
    digest.lead    ← publicLead (or empty → omits pull-quote)
    digest.signals ← publicSignals (or empty → omits signals page)
    digest.threads ← publicThreads (or category-derived stub via
                     new derivePublicThreadsStub helper — never
                     falls back to the personalised threads)
- New tests cover all three substitutions + their fail-safes.

Fix 2 — per-rule synthesis in send loop:
- Each due rule independently calls runSynthesisWithFallback over
  ITS OWN pool + ctx. Channel body lead is internally consistent
  with the storyList (both from the same pool).
- Cache absorbs the cost: when this is the winner rule, the
  synthesis hits the cache row written during the compose pass
  (same userId/sensitivity/pool/ctx) — no extra LLM call. Only
  multi-rule users with non-overlapping pools incur additional
  LLM calls.
- magazineUrl still points at the winner's envelope (single brief
  per user per slot — `(userId, issueSlot)` URL contract). Channel
  lead vs magazine lead may differ for non-winner rule sends;
  documented as acceptable trade-off (URL/key shape change to
  support per-rule magazines is out of scope for this PR).
- Parity log refined: adds `winner_match=<bool>` field. The
  PARITY REGRESSION warning now fires only when winner_match=true
  AND the channel lead differs from the envelope lead (the actual
  contract regression). Non-winner sends with legitimately
  different leads no longer spam the alert.

Test results:
- tests/brief-magazine-render.test.mjs: 75/75 (+7 new for public
  signals/threads + validator + private-mode-ignores-public-fields)
- Full data suite: 6995/6995 (was 6988; +7 net)
- typecheck + typecheck:api: clean

Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Addresses 2 review findings on PR koala73#3396 not anticipated in the
5-round Codex review.

* fix(brief): unify compose+send window, fall through filter-rejection

Address two residual risks in PR koala73#3396 (single-canonical-brain refactor):

Risk 1 — canonical lead synthesized from a fixed 24h pool while the
send loop ships stories from `lastSentAt ?? 24h`. For weekly users
that meant a 24h-pool lead bolted onto a 7d email body — the same
cross-surface divergence the refactor was meant to eliminate, just in
a different shape. Twice-daily users hit a 12h-vs-24h variant.

Fix: extract the window formula to `digestWindowStartMs(lastSentAt,
nowMs, defaultLookbackMs)` in digest-orchestration-helpers.mjs and
call it from BOTH the compose path's digestFor closure AND the send
loop. The compose path now derives windowStart per-candidate from
`cand.lastSentAt`, identical to what the send loop will use for that
rule. Removed the now-unused BRIEF_STORY_WINDOW_MS constant.

Side-effect: digestFor now receives the full annotated candidate
(`cand`) instead of just the rule, so it can reach `cand.lastSentAt`.
Backwards-compatible at the helper level — pickWinningCandidateWithPool
forwards `cand` instead of `cand.rule`.

Cache memo hit rate drops since lastSentAt varies per-rule, but
correctness > a few extra Upstash GETs.

Risk 2 — pickWinningCandidateWithPool returned the first candidate
with a non-empty raw pool as winner. If composeBriefFromDigestStories
then dropped every story (URL/headline/shape filters), the caller
bailed without trying lower-priority candidates. Pre-PR behaviour was
to keep walking. This regressed multi-rule users whose top-priority
rule's pool happens to be entirely filter-rejected.

Fix: optional `tryCompose(cand, stories)` callback on
pickWinningCandidateWithPool. When provided, the helper calls it after
the non-empty pool check; falsy return → log filter-rejected and walk
to the next candidate; truthy → returns `{winner, stories,
composeResult}` so the caller can reuse the result. Without the
callback, legacy semantics preserved (existing tests + callers
unaffected).

Caller composeAndStoreBriefForUser passes a no-synthesis compose call
as tryCompose — cheap pure-JS, no I/O. Synthesis only runs once after
the winner is locked in, so the perf cost is one extra compose per
filter-rejected candidate, no extra LLM round-trips.

Tests:
- 10 new cases in tests/digest-orchestration-helpers.test.mjs
  covering: digestFor receiving full candidate; tryCompose
  fall-through to lower-priority; all-rejected returns null;
  composeResult forwarded; legacy semantics without tryCompose;
  digestWindowStartMs lastSentAt-vs-default branches; weekly +
  twice-daily window parity assertions; epoch-zero ?? guard.
- Updated tests/digest-cache-key-sensitivity.test.mjs static-shape
  regex to match the new `cand.rule.sensitivity` cache-key shape
  (intent unchanged: cache key MUST include sensitivity).

Stacked on PR koala73#3396 — targets feat/brief-two-brain-divergence.

* feat(convex): add mcpProTokens table + issue/validate/revoke + http routes (U1)

Non-key Pro MCP identity layer for the Pro-tier MCP access plan. Mirrors
apiKeys.ts shape but stores no key material — the row's _id IS the
bearer identifier (referenced from OAuth code/token records as
mcpTokenId).

- mcpProTokens table: {userId, clientId?, name?, createdAt, lastUsedAt?,
  revokedAt?} indexed by_userId; userApiKeys untouched.
- Internal mutations: issueProMcpToken (tier ≥ 1 gate, 5-row cap with
  silent oldest-revoke rotation, audit-preserving), validateProMcpToken,
  internalRevokeProMcpToken (server-side rollback path for U5 — no Clerk
  context needed), touchProMcpTokenLastUsed (debounced 5min).
- Public mutations: revokeProMcpToken + listProMcpTokens (Clerk-auth).
- HTTP internal routes: /api/internal-{issue,validate,revoke}-pro-mcp-token
  with x-convex-shared-secret guard. Validate route schedules touch via
  ctx.scheduler.runAfter, matching apiKeys pattern (http.ts:839).

28 new tests; full convex suite 245/245.

Plan unit U1: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md

* feat(server): pro-mcp-token edge helper — issue/validate/revoke (U2)

Edge-runtime-safe wrappers around U1's Convex internal HTTP routes. No
positive cache on validate (revoke takes effect on next request);
negative-cache only at pro-mcp-token-neg:<tokenId> (60s TTL) for
known-bad bearers.

- issueProMcpTokenForUser: typed ProMcpIssueFailed (pro-required /
  invalid-user-id / config / network), 3s timeout.
- validateProMcpToken: neg-cache short-circuit BEFORE Convex hit; fail-soft
  on 5xx/timeout (returns null but does NOT write neg-cache sentinel —
  prevents transient-blip poisoning of legitimate tokens). Convex's
  validate route schedules touchProMcpTokenLastUsed internally; no
  edge-side waitUntil needed.
- revokeProMcpToken: returns result object (no throw) so rollback callers
  don't have their original error masked. Sets neg-cache sentinel after
  successful revoke.
- invalidateProMcpTokenCache: public sentinel writer for U9.

25 tests including a load-bearing integration test for revoke → next
validate short-circuits without Convex round-trip.

Plan unit U2.

* feat(oauth): apex Clerk grant page + signed-grant mint API (U3)

Bridge between the apex Clerk session and the api-subdomain Pro MCP
consent flow. Apex /mcp-grant SPA reads the OAuth nonce + registered
client metadata so users see the real client_name + redirect host
(anti-phishing). On Authorize the page POSTs to mcp-grant-mint, gets
back a fixed redirect URL with a HMAC-signed grant token, and navigates.

- mcp-grant.html + src/mcp-grant-main.ts: Vite multi-page entry (matches
  existing settings.html / live-channels.html convention).
- api/internal/mcp-grant-mint.ts: Clerk-auth POST. Re-checks tier ≥ 1,
  re-validates redirect_uri allowlist (defense-in-depth), HMAC-signs
  {userId, nonce, exp+5min}, SETEX mcp-grant:<nonce>, returns FIXED
  redirect URL (https://api.worldmonitor.app/oauth/authorize-pro).
- api/internal/mcp-grant-context.ts: GET companion for the SPA to fetch
  the real client_name + redirect_host. Same Pro-tier gate as mint.
- api/_mcp-grant-hmac.ts: shared sign/verify helper. Signature is over
  exact post-base64url-decode bytes (key-order/whitespace independent).
  U5 imports verifyGrant from this module.
- vercel.json: /mcp-grant → /mcp-grant.html rewrite + no-store header
  rule. Catch-all SPA fallback adjusted to exclude the new page.
- api/oauth/register.js: export isAllowedRedirectUri so DCR allowlist is
  reused (no parallel impl).
- 35 tests; full deploy-config + edge-functions + mcp + pro-mcp-token
  suites green.

Plan unit U3.

* feat(oauth): consent page Pro-CTA-default + 'Use API key instead' (U4)

R1 of the plan: a logged-in Pro user authorising MCP from claude.ai
never sees the API-key input field by default. The consent page now
leads with a brand-green "Sign in with WorldMonitor Pro" CTA pointing
at the apex /mcp-grant page; the existing API-key form is hidden
behind a "Use API key instead" disclosure for Starter+ holders.

- Default state: form display:none; Pro CTA dominant.
- Error state: existing errorMsg path still works — when ke.textContent
  is non-empty (or XHR-retry returns invalid_key), inline script auto-
  reveals the form so the user doesn't lose context after a bad key.
- Deep-link: /oauth/authorize?...#api-key shows the form on initial
  load, so Starter+ users can bookmark.
- Pro CTA href: https://worldmonitor.app/mcp-grant?nonce=<URL-encoded n>.
  Apex page reads the OAuth nonce server-side (no client_name forwarded
  via URL — already covered by U3's mcp-grant-context).
- nonce shared with U5: the same oauth:nonce:<n> row that U5's
  /oauth/authorize-pro will GETDEL. Handler still mints exactly one
  nonce per GET — no double-issue.
- XSS-escape preserved on client_name + redirect_host + nonce + errorMsg.
- 18 new tests; existing handler logic untouched.

Plan unit U4.

* feat(oauth): /oauth/authorize-pro bounce-back endpoint (U5)

Receives the apex grant bounce-back, validates the HMAC-signed grant,
atomically consumes both Redis nonces (mcp-grant + oauth:nonce), issues
a non-key Pro identity row in Convex via U2, and writes the OAuth code
with {kind:'pro', userId, mcpTokenId, ...} for U6 to read at exchange
time.

Security ordering (HMAC-first to avoid burning nonces on forged tokens):
  1. verifyGrant (U3's _mcp-grant-hmac, no Redis)
  2. URL-nonce vs payload-nonce match
  3. GETDEL mcp-grant:<n>
  4. strict {userId, exp} tuple match vs grant payload (forge defense)
  5. GETDEL oauth:nonce:<n>
  6. GET oauth:client:<client_id> + redirect_uri allowlist re-check
  7. getEntitlements(userId) tier ≥ 1 re-check (lapse defense)
  8. issueProMcpTokenForUser (U2)
  9. SETEX oauth:code:<code> (kind:'pro', 600s)
  10. 302 redirect_uri?code=<code>[&state=...] with Cache-Control:no-store

oauth:code shape (load-bearing for U6):
  {kind:'pro', userId, mcpTokenId, client_id, redirect_uri,
   code_challenge, scope:'mcp_pro'}

Best-effort revoke when SETEX fails after issueProMcpTokenForUser
succeeds — calls U2's no-throw revokeProMcpToken; logs orphan-row id
on revoke failure but never masks the original 500.

35 tests; vercel rewrite + api-route-exceptions registered as
external-protocol.

Plan unit U5.

* feat(oauth): McpAuthContext discriminated union + Pro token shape (U6)

Token endpoint and bearer resolver learn about Pro-shape tokens without
breaking Starter+ legacy paths.

api/_oauth-token.js:
- New resolveBearerToContext(token) returns a discriminated union:
    {kind:'env_key', apiKey}    -- legacy WORLDMONITOR_VALID_KEYS resolution
    {kind:'pro', userId, mcpTokenId}    -- new Pro identity
- resolveApiKeyFromBearer kept as backward-compat wrapper. Returns
  apiKey for env_key kind, null for pro kind (so legacy callers can't
  mis-handle a Pro bearer).

api/oauth/token.js → api/oauth/token.ts:
- Converted to TypeScript so it can import validateProMcpToken from
  server/_shared/pro-mcp-token cleanly. Vercel routes by basename;
  /oauth/token rewrite unaffected.
- Default handler wires injected deps via tokenHandler(req, deps) for
  testability (mirrors U3/U5 pattern).
- authorization_code branch: dispatches on codeData.kind. Pro path
  validates client_id + redirect_uri bind, PKCE-verifies, mints tokens
  via storeProTokens (object shape). Legacy path unchanged.
- refresh_token branch: Pro refreshes call validateProMcpToken
  (per-request Convex hit, no positive cache); revoked row → invalid_grant.
  Cross-user defensive guard: Convex-returned userId must match bearer's.
  family_id, mcpTokenId, scope ('mcp_pro') preserved across rotation.
- client_credentials grant untouched.

oauth:token:<uuid> shapes:
- Legacy (preserved): JSON.stringify("<sha256-hex-64>") or "<fingerprint-16>"
- Pro (new):          JSON.stringify({kind:'pro', userId, mcpTokenId})
- oauth:refresh:<uuid> Pro shape includes client_id, scope, family_id
  for rotation discipline (matches legacy semantics).

26 new tests; sibling suites (oauth-authorize, oauth-authorize-pro, mcp,
pro-mcp-token — 101 tests) regress green. tsc -p tsconfig.api.json clean.

Plan unit U6.

* feat(mcp): Pro identity, atomic INCR-first quota, internal HMAC fetch (U7)

Behavioral heart of the Pro MCP plan. api/mcp.ts now resolves bearers
as McpAuthContext (env_key | pro), runs Pro-specific pre-checks, and
enforces a hard 50/UTC-day cap via INCR-first reservation with
DECR-rollback on cap-exceed and on tool-dispatch failure.

Pre-dispatch chain for Pro context (synchronous, in order):
  1. validateProMcpToken(mcpTokenId)         — null = 401 revoked
  2. defensive userId match                  — 401 cross-user
  3. getEntitlements(userId) tier ≥ 1, mcpAccess: true, validUntil — fail-closed
  4. per-minute slidingWindow(60, 60s) keyed pro-user:<userId>  — fail-open
  5. for tools/call ONLY: pipeline INCR + EXPIRE 172800
       newCount > 50 → DECR rollback + -32029 + 429 + Retry-After
       INCR Redis transient → -32603 + 503 + Retry-After:5 (hard-cap correctness)
  6. dispatch tool; on throw → DECR rollback + -32603

Internal-HMAC tool fetches (replaces X-WorldMonitor-Key for Pro):
  payload   = ${ts}:${METHOD}:${pathname}:${queryHash}:${bodyHash}:${userId}
  queryHash = SHA-256(canonicalQueryString)   sorted keys, URL-encoded
  bodyHash  = SHA-256(bodyBytes)              SHA-256("") for empty
  sig       = HMAC-SHA-256(MCP_INTERNAL_HMAC_SECRET, payload)
  headers   = X-WM-MCP-Internal: ${ts}.${b64url(sig)}, X-WM-MCP-User-Id

Replay defense: payload binds method+path+queryHash+bodyHash, so a
captured signature for /api/news/v1/list-feed-digest cannot be reused
on /api/intelligence/v1/deduct-situation. ±30s window.

server/_shared/mcp-internal-hmac.ts: single source of truth for sign
helpers. U8 verify imports the same canonicalisation primitives.

server/_shared/pro-mcp-token.ts: dailyCounterKey(userId), 50/172800s
constants exported for U9 to read the SAME Redis key the enforcement
writes.

_execute signatures changed (params, base, context) — option A from
the plan. Cache-only tools unchanged.

waitUntil unused: Convex internal-validate-pro-mcp-token schedules
touchProMcpTokenLastUsed itself (verified at convex/http.ts:1035-1040
from U1's commit 4530bdf).

22 new tests + 23 Starter+ regression tests in tests/mcp.test.mjs;
sibling chain (oauth-token, pro-mcp-token, oauth-authorize-pro,
mcp-grant-mint, mcp-proxy) all green. tsc + biome clean.

Plan unit U7.

* feat(gateway): HMAC-verify internal-MCP requests + sanitised propagation (U8)

Verifier counterpart to U7's signer. Gateway accepts X-WM-MCP-Internal
HMAC headers in lieu of an API key for Pro internal-MCP traffic, then
hands a freshly-constructed Request with sanitised trusted markers to
the downstream handler. isCallerPremium learns to honour those markers
so Pro framework/systemAppend semantics survive in summarize-article,
get-country-intel-brief, deduct-situation.

Gateway flow at the top of createDomainGateway's handler:
  1. STRIP inbound x-wm-mcp-internal-verified + x-user-id (anti-injection)
  2. If X-WM-MCP-Internal present:
       verifyInternalMcpRequest re-canonicalises and HMAC-verifies the
       request shape (method+pathname+queryHash+bodyHash+userId), ±30s
       window, timing-safe compare. Failure → 401 invalid_internal_mcp_
       signature (no fall-through to validateApiKey — present-but-bad is
       a deliberate forge attempt).
       getEntitlements(userId): tier ≥ 1 + mcpAccess === true (fail-closed).
       Construct NEW Request via new Request(url, {method, body, headers})
       with x-wm-mcp-internal-verified=<per-process nonce> +
       x-user-id=<verified userId>. Skip validateApiKey + IP rate limit.

isCallerPremium adds a NEW first branch: timing-safe compare of
x-wm-mcp-internal-verified against the per-process nonce + non-empty
x-user-id + defensive getEntitlements re-fetch. Catches direct-edge-
function consumers (api/widget-agent.ts, chat-analyst.ts, me/entitlement.ts,
v2/shipping/webhooks/...) that don't run the gateway strip step.

DEFENSE-IN-DEPTH BEYOND PLAN: trusted marker is a per-process random
16-byte nonce, NOT the literal '1'. Sweep found direct edge functions
calling isCallerPremium without gateway-strip protection — a constant
marker would be spoofable from outside. The nonce is born once at
process startup, only the gateway knows it, comparison is timing-safe.

verifyInternalMcpRequest lives in server/_shared/mcp-internal-hmac.ts
next to U7's sign helpers — single source of truth for canonical form
+ payload + ±30s window. Drift between sign and verify is structurally
impossible.

26 new tests + 100 sibling regression tests (gateway-cdn-origin-policy,
premium-stock-gateway, premium-fetch, pro-mcp-token, mcp). tsc clean.

Plan unit U8.

* feat(catalog): mcpAccess feature flag + env vars + Pro-MCP docs (U10)

Catalog, schema, type, env, and docs scaffolding to make the Pro MCP
flow deployable end-to-end. U9 unblocks on this — settings UI gates on
hasFeature('mcpAccess') (distinct from existing apiAccess paywall).

- convex/config/productCatalog.ts: PlanFeatures.mcpAccess on every tier.
  Free=false. Pro_monthly/Pro_annual=true. API_starter / API_starter_annual
  / API_business=true. Enterprise=true. Pro plan marketing copy mentions
  "MCP access for Claude Desktop & other AI clients (50 calls/day)".
- convex/schema.ts + convex/payments/cacheActions.ts: mcpAccess optional
  field on the entitlements validator; legacy rows pass.
- server/_shared/entitlement-check.ts: CachedEntitlements.features
  mcpAccess?: boolean (consumed by U7 + U8).
- src/services/entitlements.ts: EntitlementState.features.mcpAccess?:
  boolean — unblocks hasFeature('mcpAccess') for U9's settings tab gate.
- .env.example: new "Pro MCP" section with MCP_PRO_GRANT_HMAC_SECRET
  (apex grant bridge, U3/U5) and MCP_INTERNAL_HMAC_SECRET (gateway
  service-auth, U7/U8). Both 32-byte base64; both required.
- docs/mcp-server.mdx: "Pro sign-in flow" section + 50/day quota
  callout + "Connected MCP clients" management pointer.

Bundled fixups for U7/U8 strict-mode TS errors:
- mcp-internal-hmac.ts: bufferToBase64Url uses for-of (avoids
  noUncheckedIndexedAccess error on bytes[i]).
- gateway.ts: drop unused INTERNAL_MCP_USER_ID_HEADER import.

tsc -p tsconfig.api.json + tsc -p tsconfig.json clean.
9 entitlements tests + 71 gateway-internal-mcp + mcp tests pass.

Plan unit U10.

* feat(settings): Connected MCP clients tab + quota endpoint + revoke (U9)

User-facing surface that makes the Pro MCP plan visible. Settings UI
gets a new "Connected MCP clients" tab gated on hasFeature('mcpAccess')
(distinct from the existing apiAccess-gated 'API Keys' tab — Pro users
without apiAccess see only the new tab).

Endpoints:
- GET /api/user/mcp-quota → {used, limit:50, resetsAt}. Reads the
  SAME Redis key (dailyCounterKey from U2) that U7 enforcement writes.
  Single source of truth.
- POST /api/user/mcp-revoke {tokenId} → forwards Clerk-derived userId
  (NOT client-supplied) to Convex internal-revoke-pro-mcp-token, then
  calls invalidateProMcpTokenCache so any in-flight bearer with this
  tokenId 401s within 60s. 404 on NOT_FOUND, 409 on ALREADY_REVOKED.
  Tenancy enforced inside Convex (row.userId === userId).

UI (in src/components/UnifiedSettings.ts, no child component split —
matches existing API Keys tab pattern):
- Tab gated on hasFeature('mcpAccess') so Pro users see it without
  needing apiAccess.
- Quota header auto-refreshes every 30s; interval cleared on tab-
  switch / close / destroy.
- Each row: name, createdAt, lastUsedAt (relative), revokedAt
  (struck-through + badge); active rows show Revoke button with
  confirm() dialog (matches existing pattern).
- Empty state: click-to-copy https://api.worldmonitor.app/mcp.

Frontend service in src/services/mcp-clients.ts: listMcpClients (Convex
query), revokeMcpClient (POST /api/user/mcp-revoke), fetchMcpQuota.

11 + 14 = 25 new tests; api-route-exceptions registered as
internal-helper.

Plan unit U9.

---

PLAN COMPLETE — 10/10 units shipped. Follow-up polish noted in U9
report: CSS rules for .mcp-clients-* classes (suggest copy from
.api-keys-* rules); empty-state URL is hardcoded to
api.worldmonitor.app/mcp.

* style(settings): mcp-clients CSS rules mirroring api-keys (U9 polish)

* fix(pro-mcp): apply Tier-2 code-review residuals — security + correctness

Review pass after U10 (3 reviewers: code-reviewer, ce-security-reviewer,
ce-adversarial-reviewer). Findings BLOCKING + HIGH + MEDIUM + LOW
applied per user direction.

BLOCKING (gateway):
- Add validUntil check to gateway's HMAC-verify entitlement re-check.
  Defense-in-depth against Convex-fallback paths returning a stale row
  past its expiry.

HIGH — adv-001: cross-user OAuth nonce hijack:
- mcp-grant-mint now claims oauth nonces atomically via Upstash SET NX
  semantics. If the nonce is already claimed by a DIFFERENT userId,
  return 403 NONCE_CLAIMED_BY_OTHER_USER. Same userId multi-tab retry
  is idempotent: re-sign the grant using the existing claim's exp so
  authorize-pro's strict tuple equality still matches.
- mcp-grant-context performs the same cross-user check so the apex SPA
  refuses to render consent context for a hijacked nonce.

MEDIUM — adv-008: refresh-token loss during Convex transient:
- validateProMcpToken returns a discriminated union {ok:'valid',userId}
  | {ok:'revoked'} | {ok:'transient'}. Negative-cache writes only on
  'revoked'. validateProMcpTokenOrNull wrapper preserves backward
  compat for callers that don't need the distinction (per-request MCP
  edge stays fail-closed).
- token.ts refresh path: on 'transient', best-effort restore the
  consumed refresh token to Redis with the original TTL and return
  503 server_error + Retry-After:5. Client retries; if Convex recovers,
  refresh succeeds. No more permanent session loss on a Convex blip.

MEDIUM — adv-002: counter overshoot lockout:
- mcp.ts: on cap-exceeded after DECR rollback, if newCount is still
  far above PRO_DAILY_QUOTA_LIMIT (overshoot from prior transient),
  pipelined INCR+DECR probe + bounded DECR-sweep to converge the
  counter back near the limit. Prevents one Redis hiccup from locking
  out a paying Pro user for the rest of the UTC day.

MEDIUM — code-reviewer #4: 5-row cap race in Convex:
- mcpProTokens.issueProMcpToken now revokes ALL active rows beyond
  MAX-1 (sorted by createdAt) on each issue, so concurrent inserts
  that briefly produce 6 active rows converge back to 5 on the next
  call.

LOW:
- adv-003: executeTool throws cache_all_null when every cache read
  returns null AND there were keys configured — triggers DECR rollback
  in dispatchToolsCall instead of silently burning quota on a degenerate
  empty result.
- code-reviewer koala73#10: gateway strips X-WM-MCP-Internal + X-WM-MCP-User-Id
  from the trusted Request before forwarding to handler (no info leak).
- adv-004: 256 KB body cap (Content-Length + post-buffer count) on
  both gateway strip and HMAC-verify paths — bounds memory amplification.
- adv-006: dailyCounterKey now env-prefixes (preview deploys can't
  collide with production traffic on the same Upstash instance).
  Reader (mcp-quota.ts) and writer (mcp.ts) share byte-identical keys
  via the helper.
- adv-007 / code-reviewer #5: signInternalMcpRequest throws on
  Blob/FormData/ReadableStream/plain-object bodies instead of silently
  JSON.stringify'ing them (which would produce a hash that can't match
  the wire bytes).
- code-reviewer koala73#9: timestamp regex tightened to ^[0-9]{1,15}$ so
  future ms-precision timestamps don't silently truncate through Number().
- code-reviewer koala73#8: MCP_INTERNAL_HMAC_SECRET preflight in runProPreChecks
  surfaces config errors at auth-resolution rather than mid-tool-fetch.

NITPICK:
- code-reviewer #6: rewritten readNegCache comment.
- code-reviewer #7: malformed-body error now reports reason
  'malformed_request' instead of misleading 'auth_401'. New
  RequestReason value added to usage.ts enum.

30 new tests + ~14 adapted; full convex suite 247/247, edge/server
suites 254/254. tsc clean across both tsconfig.json and tsconfig.api.json.

Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md

* chore(pro-marketing): regenerate /pro bundle for U10 catalog copy

U10's productCatalog.ts added an mcpAccess feature flag and updated the
Pro plan description copy to "MCP access for Claude Desktop & other AI
clients (50 calls/day)". scripts/generate-product-config.mjs cascades
that copy into pro-test/src/generated/tiers.json, which the /pro
marketing app builds into public/pro/assets/.

Vercel does NOT rebuild pro-test on deploy — public/pro/ ships whatever
is committed. Pre-push hook auto-ran the build and produced a new bundle
hash; this commit lands the regenerated artifacts so deploy picks them up.

Plan unit U10 (follow-on artifact).

* fix(pro-mcp): apply external review round-2 findings (P1+P2+P3)

Round-2 review on PR koala73#3646 surfaced 5 must-fix issues:

P1 — vercel.json `(?:...)` rejected by Vercel CI:
- Replace `mcp-grant(?:\\.html)?` with explicit `mcp-grant\\.html|mcp-grant`
  inside the SPA catch-all negative-lookahead (lines 18 + 134).
- Split the headers `/mcp-grant(?:\\.html)?` source rule into two explicit
  entries — Vercel's path-to-regexp `source` field doesn't accept `(?:...)`
  with `?` quantifier as a standalone source.
- Update tests/deploy-config.test.mjs expectations to match.

P1 — api/api-route-exceptions.json points at deleted file:
- Update the entry path from `api/oauth/token.js` (deleted in U6) to
  `api/oauth/token.ts`. `node scripts/enforce-sebuf-api-contract.mjs`
  was failing pre-push CI.

P1 — mcpAccess migration gap on existing entitlement rows:
- convex/entitlements.ts now read-time merges `getFeaturesForPlan(planKey)`
  defaults under stored features. Pre-U10 rows lacking `mcpAccess` see the
  catalog default surfaced immediately, with NO wait for the next Dodo
  webhook to rewrite the row. Stored features still win on conflict
  (preserves per-user overrides). Mirrored explicitly in
  convex/mcpProTokens.ts:55 (issueProMcpToken's direct ctx.db.query path
  computes the same merge inline since it bypasses the query handler).

P2 — grant/issue path missing mcpAccess gate (let tier-1-without-mcpAccess
users complete OAuth then fail every tools/call at the gateway):
- mcp-grant-context.ts:95, mcp-grant-mint.ts:236, authorize-pro.ts:356,
  convex/mcpProTokens.ts:55 now ALL gate on `tier ≥ 1 && mcpAccess === true`,
  matching the downstream MCP-edge runProPreChecks check. Widens the
  `getEntitlements` deps return type at all three handlers.

P3 — inline theme script blocked by global CSP:
- mcp-grant.html:34 inline `<script>` is removed (global CSP at vercel.json:92
  is hash-allowlist-based and we don't allowlist this page's hashes).
- Theme application moved into the bundled module at src/mcp-grant-main.ts
  (top of file, runs on module evaluation). Brief default-theme flash on
  light-preference users is acceptable for a transient consent UI.

10 new tests covering: mcpAccess: false at each of the 4 gates,
mcpAccess: undefined (legacy row) at each of the 4 gates, and the
read-time catalog merge in both directions (legacy gets default,
override preserved).

Full gate post-fix:
- Convex 249/249 (+2 new merge tests).
- Edge/server 280/280 (+10 new gate tests).
- tsc both configs clean.
- sebuf contract clean (was failing before P1.2).
- Sentry coverage clean.
- deploy-config tests assert new explicit-source rules.

Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md

* fix(entitlement-cache): treat pre-U10 cache entries lacking mcpAccess as stale

Reviewer round-2 P2 (cache layer): _getEntitlementsImpl returned hot
Redis cache entries as-is, bypassing the read-time catalog merge that
convex/entitlements.ts:50 applies on the Convex read path. Paying users
with cache entries written before plan 2026-05-10-001 U10 deployed see
mcpAccess !== true at the grant/MCP gates and are blocked for up to
the 15-min cache TTL.

Cache predicate now also requires `typeof features.mcpAccess === 'boolean'`.
Legacy entries (mcpAccess: undefined) fall through to Convex, which
returns the merged shape and the cache is rewritten with the post-U10
layout. Self-healing, bounded to one extra Convex round-trip per
affected user during the migration window.

Targeted choice over alternatives:
- Importing convex/config/productCatalog into server/_shared to apply
  the merge at the cache layer would pull non-edge-safe deps. Rejected.
- Treating ALL cached entries as stale would defeat the cache layer.
  Rejected.
- typeof check on the new field is the minimum delta that fixes the
  migration window without restructuring.

Test fixture in server/__tests__/entitlement-check.test.ts updated:
makeEntitlements now includes mcpAccess (tier ≥ 1 → true). Two new
focused tests:
1. Legacy cache entry without mcpAccess → cache rejected → Convex
   round-trip → result returned (verifies fetch called once).
2. Post-U10 cache entry WITH mcpAccess → cache honored → no Convex
   round-trip (verifies fetch called zero times).

Convex 251/251 (+2 new cache tests).

Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md

…9 regression) (koala73#3835)

The 2026-05-19 Pro brief shipped "How nuclear war would impact the
global food system" from Bulletin of Atomic Scientists as CRITICAL
story #6, sitting alongside breaking news about the Iran-Israel war
and the WHO Ebola declaration. The brief promises event-driven
intelligence — a Bulletin analysis essay is not an event.

The existing classifier missed it on all three current signals:
- STRONG #1 (URL section /opinion/): Bulletin URLs have no opinion-
  style path because the WHOLE SITE is commentary, no hard-news
  section to distinguish from.
- STRONG #2 (headline prefix "Opinion:"): hard-news-shaped title.
- CORROBORATING (quote-wrap + columnist framing): description reads
  like a news lede.

The signal IS the publisher. Add STRONG #3: a hand-curated allowlist
of commentary-only publishers, matched by hostname (suffix-anchored to
permit `newsletter.<host>` / `m.<host>` while rejecting typo-domains
like `evilthebulletin.org`).

Initial list (per docs/plans/2026-05-19-001 U1):
- thebulletin.org      — Bulletin of Atomic Scientists
- project-syndicate.org — op-eds from world leaders / academics
- foreignaffairs.com   — CFR's analysis quarterly
- foreignpolicy.com    — Foreign Policy magazine
- warontherocks.com    — defense analysis blog

Maintenance commitment: quarterly review against `droppedOpinion`
telemetry to catch (a) new commentary publishers to add, (b) listed
publishers that launched a hard-news section. Rollback path is
remove-from-Set, never per-URL exceptions (cruft).

Pattern mirrors the existing safePathname/STRONG_URL_SEGMENTS shape;
new safeHostname helper parses URL().hostname so the match is on
the parsed host (not raw .includes() on the link string — closes the
tracking-param injection vector documented in PR koala73#3748).

7 new test cases cover the May 19 regression, the 4 other publishers,
subdomain matching, typo-domain rejection, tracking-param / fragment
spoofing, malformed URLs, and the "allowlisted host PLUS hard-news
content → still opinion" rule.

This is PR-1 of the 4-PR Phase 1 wave from
docs/plans/2026-05-19-001-fix-brief-content-quality-regressions-plan.md.