…ecovery (koala73#3476)

* fix(broadcast): lease-based race guard + structured partial-failure recovery

Addresses two P1 review findings on PR koala73#3473 (rampRunner already merged).

P1 #1 — Race condition that allowed DUPLICATE EMAIL SENDS

The runner read currentTier and ran assignAndExportWave +
createProLaunchBroadcast + sendProLaunchBroadcast BEFORE recording the
tier in _recordWaveSent. Two overlapping cron runs (or cron + manual
trigger, or Convex action retry) both passed kill-gate / tier checks,
both proceeded through all 3 external side effects, and only collided
at _recordWaveSent — by then duplicate emails had gone out to every
recipient in the wave. The tier check there is post-hoc, not a pre-claim.

Fix: atomic lease, claimed BEFORE any external side effect. New mutation
`_claimTierForRun(runId, expectedCurrentTier)` writes pendingRunId +
pendingRunStartedAt iff:
- currentTier matches expected
- no fresh lease is already held (or held lease is older than
  STALE_LEASE_MS = 30min, in which case override the abandoned lease
  from a crashed runner)
The runner exits cleanly on claim rejection without ever touching
external state. _recordWaveSent now validates the lease still belongs
to its runId and clears it on success. _recordRunOutcome / clearPartialFailure
also clear the lease (only when runId matches, to avoid stomping
another run's claim).

P1 #2 — Recovery path broken when assignAndExportWave succeeded but
createProLaunchBroadcast or sendProLaunchBroadcast failed

In that intermediate "stamped + segment created + not sent" state, bare
clearPartialFailure makes the next cron retry the SAME waveLabel, which
fails because contacts are already stamped. The cron thrashes on the
same partial-failure forever short of abortRamp or hand-patching the DB.

Fix: new `recoverFromPartialFailure(recovery, ...)` action with two
explicit operator-declared modes:
- manual-finished: operator manually completed the wave (e.g. via Resend
  dashboard or direct sendProLaunchBroadcast call). Pass broadcastId,
  segmentId, sentAt, assigned. Tier advances; next kill-gate uses the
  recorded broadcastId. Result equivalent to a successful cron run.
- discard-and-rotate: write off the lost wave. Bumps waveLabelOffset by
  1 so the next cron uses a FRESH waveLabel; the prior label's stamps
  remain in the audience table and exclude those contacts from future
  picks (lost to this campaign; operator can manually email later).
  Tier NOT advanced (no successful send to record).

Both modes clear the lease and the partial-failure status. The runner's
partial-failure error messages now include the recoverFromPartialFailure
invocation hint so on-call ops can see the recovery path right next to
the error.

Schema additions (convex/schema.ts): pendingRunId, pendingRunStartedAt
optional fields on broadcastRampConfig. getRampStatus surfaces both for
operator visibility, plus a derived `leaseHeld` boolean.

Tests (convex/__tests__/rampRunner.test.ts): 13 cases covering
- _claimTierForRun: fresh claim, lease-held rejection (THE RACE GUARD),
  tier-moved rejection, stale-lease override
- _recordWaveSent: lease-validated success, lease-lost rejection
- _recordRunOutcome: clears own lease, leaves other lease alone
- recoverFromPartialFailure: manual-finished advances tier with operator-
  provided broadcastId, manual-finished requires all wave fields,
  discard-and-rotate bumps waveLabelOffset and DOES NOT advance tier,
  noop when status not partial-failure
- end-to-end: two concurrent claims, exactly one wins

107/107 total Convex tests pass (was 94; +13). TS dual typecheck clean,
biome clean.

* fix(broadcast): drop stale-lease auto-override + persist per-step progress + harden recovery (PR koala73#3476 review round 2)

Addresses 4 P1 findings on PR koala73#3476:

P1#1 (rampRunner.ts:455-474) — Stale-lease auto-override removed.
A 30-min wall-clock cutoff has the same failure mode the lease exists to
prevent: assignAndExportWave can legitimately exceed the cutoff for large
waves / slow Resend, and overriding while the original run is alive lets a
second run race and duplicate-send. A held lease now blocks until cleared
by the owning run (terminal outcome path) or by the operator via
forceReleaseLease (last-resort, sets lastRunStatus=partial-failure so
recoverFromPartialFailure can pick up).

P1#2 (rampRunner.ts:545-575) — _recordRunOutcome lease-mismatch is a hard
no-op. Previously it suppressed the lease-clear but still patched
lastRunStatus / lastRunError / killGateTripped / active when runId did not
match pendingRunId, clobbering the winner's authoritative outcome.

P1#3 (rampRunner.ts:236-249) — clearPartialFailure now requires
`confirmNoExport: v.literal(true)` AND fail-closes if any pending* progress
marker is set (which the runner persists as soon as any external step
succeeds). The risky path — clear-then-retry after a stamped/sent wave —
is rejected loudly; operator must use recoverFromPartialFailure instead.

P1#4 (rampRunner.ts:276-355) — Per-step progress persistence.
_recordPendingExport (after assignAndExportWave) and
_recordPendingBroadcast (after createProLaunchBroadcast) write
(waveLabel, segmentId, assigned, broadcastId) to the config row. When the
action dies between steps (Convex action timeout, OOM) before any catch
block runs, recoverFromPartialFailure falls back to the persisted state
instead of requiring the operator to reconstruct from the Resend
dashboard. Operator-supplied args still take precedence; sentAt remains
operator-only (no marker captures send-completion time).

Schema: adds pendingWaveLabel, pendingSegmentId, pendingAssigned,
pendingExportAt, pendingBroadcastId, pendingBroadcastAt to
broadcastRampConfig (all optional). _recordWaveSent clears all six on
success; recoverFromPartialFailure clears them on completion.

Tests: ramp runner suite goes 13 → 29. Inverted stale-lease test (now
asserts rejection); added forceReleaseLease lifecycle, clearPartialFailure
fail-closed-on-pending, _recordPendingExport/Broadcast lease validation,
_recordWaveSent clears pending*, manual-finished auto-fills from
persisted state, sentAt-still-required, operator-override-beats-fallback,
discard-and-rotate clears pending*.

* fix(broadcast): persist pending export markers BEFORE inspecting failure counters (PR koala73#3476 review round 3)

P1 from review round 3: `_recordPendingExport` was called only after the
`(failed > 0 || stampFailed > 0)` and `underfilled` branches recorded
partial-failure and bailed. Those branches DID see a real export — segment
created, contacts may be stamped, contacts may be in the Resend segment —
but with the persistence call deferred past them, the row's `pending*`
markers stayed undefined. An operator running `clearPartialFailure({
confirmNoExport: true })` would then NOT trip the fail-closed guard
(because the marker check sees no `pendingWaveLabel/SegmentId/BroadcastId`),
silently masking the stamped contacts.

Fix: move `_recordPendingExport` immediately after `assignAndExportWave`
returns successfully, BEFORE inspecting `failed`, `stampFailed`, or
`underfilled`. The export ran — record that fact unconditionally on
success, regardless of whether the result counters then route us to a
partial-failure outcome. Lease validation in `_recordPendingExport`
preserves the operator-force-release safety.

Test: source-grep regression test asserts `_recordPendingExport,` call site
in `runDailyRamp` precedes every `exportResult.failed`/`stampFailed`/
`underfilled` source occurrence. Catches any future reordering that would
re-open the gap.

* fix(broadcast): route pool-drained through recoverable partial-failure (PR koala73#3476 review round 4)

P1 from review round 4: the underfilled branch was recording status=
"pool-drained" + deactivate + clearing the lease. By that point
assignAndExportWave had already stamped contacts AND created the Resend
segment AND pushed contacts — but no broadcast was ever created or sent.
The contacts are now excluded from future picks (stamped) but receive no
email; recoverFromPartialFailure can't help because status is
"pool-drained" not "partial-failure".

Fix: route through the recoverable partial-failure path. status=
"partial-failure" + deactivate=true + the persisted pending* markers
(set by _recordPendingExport in round 3) make
recoverFromPartialFailure({recovery:"manual-finished", sentAt:<epoch>})
able to pick up exactly where the runner stopped — operator manually
completes the broadcast in Resend, then advances the tier with
zero-reconstruction recovery. Or recoverFromPartialFailure({recovery:
"discard-and-rotate"}) to explicitly abandon the wave (operator's call,
not the runner's).

Operator-facing return status renamed to "pool-drained-partial-failure"
so logs distinguish this case from a generic partial-failure.

Test: source-grep regression test asserts the underfilled branch records
status:"partial-failure" (not "pool-drained"), keeps deactivate:true, and
includes recoverFromPartialFailure guidance in the error message.