fix(payments): reconcile missed Dodo renewals#4794
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| ); | ||
|
|
||
| crons.daily( | ||
| "dodo-renewal-reconciliation", | ||
| { hourUTC: 3, minuteUTC: 17 }, | ||
| internal.payments.billing.reconcileMissedDodoRenewals, | ||
| {}, |
There was a problem hiding this comment.
No follow-up scheduling when
hasMore: true
The cron always fires with {}, which uses the hard cap of 50. When stale.length === limit the action returns hasMore: true, but there is no mechanism to schedule a second pass within the same day. During a Dodo webhook outage that lets hundreds of subscriptions go stale, only the 50 oldest are reconciled each calendar day — users beyond position 50 whose subscriptions were actually cancelled or expired in Dodo will continue to receive entitlements they are no longer entitled to until their slot is reached on a future day's run.
|
|
||
| const normalized = normalizeRemoteSubscription(remote); | ||
| if (normalized.kind === "unsupported-status") { | ||
| summary.skipped++; | ||
| console.warn( | ||
| `[billing/reconcile] Skipping subscription ${normalized.dodoSubscriptionId}: unsupported Dodo status "${normalized.status}"`, | ||
| ); | ||
| continue; |
There was a problem hiding this comment.
Unsupported-status subscriptions permanently hold batch slots
When Dodo returns a status value outside active | on_hold | cancelled | expired (e.g., if Dodo introduces a new lifecycle state), normalizeRemoteSubscription returns kind: "unsupported-status", the action increments skipped, and the local record is left untouched — status = "active", currentPeriodEnd still in the past. Because the record is never updated, it will reappear in every subsequent daily cron run, consuming one of the 50 available batch slots and generating a Dodo API call, indefinitely. A subscriber stuck in this state also retains their entitlement forever since isCoveringAt treats status === "active" as always-covering regardless of period end.
…round 2)
Fold in the reliability + test findings from the ce-code-review pass:
- Bound the Dodo poll (`payments.retrieve`) with `{ timeout: 10s,
maxRetries: 1 }`. The SDK default is 60s x 2 retries (~180s); on a Dodo
outage one degraded poll could otherwise stall the whole sequential
batch — the exact failure this cron exists to survive. Mirrors the
Resend timeout already added.
- Run the reconciliation cron every 6h instead of daily. A payment
becomes a candidate at ~6h pending, so on a daily cadence its age at
first scan is uniformly 6h-30h and anything in (24h, 30h] misses the
24h customer-email freshness gate — ~25% of stuck payments silently
dropped to ops-only. At 6h cadence first-scan age stays <=~12h. Safe:
the action is idempotent and marker-gated.
- Dropped-webhook guard: when a reconciled SUCCEEDED payment carries a
subscription id but has no subscriptions row, console.error (Sentry).
A dropped payment webhook may travel with a dropped subscription.active
webhook — charged customer, no entitlement, and #4794's renewal
reconciler only scans active rows, so nobody else catches it.
- Correct the scan-cap comment/message: newest-first means over-cap OLD
rows are deferred indefinitely (until the backlog clears), not "to a
later run".
- Tests: action-level unrecognised-status (requires_payment_method) path,
two-run at-most-once-email idempotency, and the dropped-subscription
guard (pages / does not page).
Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
* feat(payments): reconcile stuck pending payments * fix(payments): harden stuck-pending reconciliation (PR #4795 review) Address the adversarial review of the stuck-pending payment cron: - F1: collapse unrecognised non-terminal Dodo IntentStatus (incl. null / requires_payment_method — the typical abandoned-3DS end-state) to still-pending, writing a marker that records the raw observed status in every fall-through. Previously these returned unknown_status with NO marker → daily re-poll for 14 days + batch-slot starvation. - F2: page ops for real. The still-pending ops branch now console.errors (Convex auto-Sentry, refund-alert precedent) instead of a never-surfaced console.warn. - F3: claim the reconciliation marker BEFORE sending the customer email (idempotency barrier — a post-send failure can't re-email next day), finalize after; split failure reporting by phase (poll / email / record). - F4: gate the "continue checkout" email on link freshness (<24h). Older candidates route to ops rather than emailing a dead Dodo checkout link. - F5: scan newest-first (.order desc) so freshly-stuck rows don't fall off the 500-row scan cap; log when the cap is hit. - F6: guard ctx.scheduler.runAfter with its own try/catch; add AbortSignal.timeout to the Resend POST; stop logging the raw Resend response body (can echo the recipient email). - F7: add reconciliation tests (unknown/null status marker, marker-before- email ordering, already_terminal race, Resend-throw-mid-batch isolation, scan-cap ordering). Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ * fix(payments): reconciliation reliability hardening (PR #4795 review round 2) Fold in the reliability + test findings from the ce-code-review pass: - Bound the Dodo poll (`payments.retrieve`) with `{ timeout: 10s, maxRetries: 1 }`. The SDK default is 60s x 2 retries (~180s); on a Dodo outage one degraded poll could otherwise stall the whole sequential batch — the exact failure this cron exists to survive. Mirrors the Resend timeout already added. - Run the reconciliation cron every 6h instead of daily. A payment becomes a candidate at ~6h pending, so on a daily cadence its age at first scan is uniformly 6h-30h and anything in (24h, 30h] misses the 24h customer-email freshness gate — ~25% of stuck payments silently dropped to ops-only. At 6h cadence first-scan age stays <=~12h. Safe: the action is idempotent and marker-gated. - Dropped-webhook guard: when a reconciled SUCCEEDED payment carries a subscription id but has no subscriptions row, console.error (Sentry). A dropped payment webhook may travel with a dropped subscription.active webhook — charged customer, no entitlement, and #4794's renewal reconciler only scans active rows, so nobody else catches it. - Correct the scan-cap comment/message: newest-first means over-cap OLD rows are deferred indefinitely (until the backlog clears), not "to a later run". - Tests: action-level unrecognised-status (requires_payment_method) path, two-run at-most-once-email idempotency, and the dropped-subscription guard (pages / does not page). Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
…e statuses, ordering) Address adversarial review of the missed-Dodo-renewal reconciler: - Poison-row starvation: record per-row reconcile attempts (lastReconcileAttemptAt + reconcileFailureCount) with exponential backoff so a permanently-failing row (e.g. test-mode-era sub 404ing the live client) stops re-occupying the ordered scan window every run. Widen the candidate scan, filter backed-off rows, and continuation-schedule the remainder within the cron cycle (bounded by MAX_CONTINUATIONS). - Remote failed/pending: map Dodo `failed` -> local `expired` (entitlement recompute downgrades instead of granting forever via isCoveringAt); skip `pending`/unknown explicitly and escalate the skip log to console.error so Convex auto-Sentry captures it. - Ordering guard: add the webhook handlers' isNewerEvent(existing.updatedAt, observedAt) check so a concurrent subscription.plan_changed write is not clobbered by the cron's stale read. - Runtime cap: bail the loop on a wall-clock budget well under Convex's 10-min action cap and continuation-schedule the rest. - DRY: reuse the exported resolvePlanKey webhook resolver instead of the duplicated productPlans -> alias -> enterprise-fallback chain. - Tests: enterprise fallback, remote-cancelled downgrade, remote-failed -> expired, both skip guards, ordering guard, unsupported/pending skips, and the starvation/attempt-marker + continuation drain behavior. Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
…empt marks Address ce-code-review findings (performance P2×2/P3, reliability P1, testing P1/P2, adversarial P2, maintainability P2) on the reconciler: - Saturation stall (P1, cross-model + perf): the fixed 500-row scan filtered backoff eligibility in memory, so a window fully within its cooldown left eligible=[] with attempted=0 and scheduled no continuation — silently stranding healthy stale rows behind it. Thread a currentPeriodEnd cursor through continuations: advance past a drained/saturated window (guaranteeing forward progress even when attempted=0) and emit a distinguishable console.error when a window is cooldown-saturated. Adds a clamped `scanLimit` arg (ops knob + deterministic test seam). - Single-transaction backoff (perf P2): fold the attempt bookkeeping into applyDodoSubscriptionReconciliation's still-stale skip branches instead of a second markDodoReconcileAttempt round-trip; reorder guards so the concurrent skips are known stale-active. markDodoReconcileAttempt now only serves the paths that never reach apply (Dodo lookup failure, unsupported/pending). - Safe attempt marking (reliability P1): the bookkeeping mutation now swallows and logs its own errors so an OCC conflict can never abort the batch or skip continuation scheduling. - Faster first-failure retry (adversarial P2): first failure backs off only a few hours (skips the rest of the cycle but retries at the next daily run) so a transient Dodo error does not over-delay a legitimate downgrade; repeated failures still back off exponentially. - Stricter ReconciliationSkipReason union; corrected continuation-ceiling comment; documented residuals. - Tests: time-budget bail (Date.now spy), first-failure-then-next-day backoff, cursor-advance past a saturated window, and a load-bearing ordering-guard assertion (seeded productPlans so a removed guard would fail it). Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
…e row Consolidate the remaining ce-code-review findings on the renewal reconciler: - Reliability P1 (no-SDK-retry): getDodoClient now uses maxRetries: 0. The SDK retry path honors a server Retry-After verbatim and could sleep minutes, blowing Convex's 10-min action cap; row-level backoff + continuation is the retry mechanism, so each lookup stays bounded by the 10s timeout. - Reliability P1 (safe backoff write): the best-effort attempt marking is now a swallow-and-log helper (safeMarkReconcileAttempt), so an OCC conflict on the bookkeeping mutation can never propagate out of the loop and abort the batch or skip continuation scheduling. - Correctness/adversarial (404 -> expired): distinguish a definitive Dodo NotFound (status 404) from transient errors. After a confirmation (the row already has >= 1 recorded failure, i.e. a 2nd consecutive 404) the row is downgraded to expired via expireMissingDodoSubscription + recompute, so a permanently-gone poison row LEAVES the active set — resolving the scan-slot occupation and the entitlement over-grant together. A single flaky 404 or any transient 5xx/network/timeout stays on the backoff path (never downgrades). - Correctness P3: a reconcile that leaves the row still stale (remote active, period end still past) records a backoff instead of clearing bookkeeping, avoiding a redundant re-lookup next cycle. - Maintainability P2: extract the per-row fetch->normalize->apply->interpret body into reconcileOneStaleRow; dedupe the status union (export SubscriptionStatus from subscriptionHelpers). - P3: one-line comment acknowledging the intentional no-run-lock (isNewerEvent + OCC self-heal overlaps). - Tests: 404-twice -> expired + single-404 stays active; 5xx never downgrades; safeMarkReconcileAttempt swallows a throwing mutation; failureCount>=2 (2-day) backoff eligibility. Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
… paging
Add the required mass-404 circuit breaker and fix two P1s my own adversarial
re-review caught in the prior fix commit:
- Mass-404 circuit breaker (required): cap confirmed-not-found downgrades per
invocation at min(5, ceil(plannedAttempts/2)). Beyond it, stop downgrading,
console.error("mass Dodo 404s … possible wrong-environment/API-key misconfig;
further downgrades halted this run"), and route the rest to backoff. A genuine
handful of deleted subs still gets cleaned; a config error that 404s the whole
base downgrades at most the threshold before self-halting (and pages ops).
- Consecutive-404 confirmation (re-review P1): the terminal-404 gate used the
generic reconcileFailureCount, so a prior unrelated failure (5xx/timeout) plus
a single 404 could downgrade a live customer. Add a dedicated
reconcileNotFoundCount that only advances on a definitive 404 and resets on any
non-404 outcome, and clear all reconcile bookkeeping in the webhook
active/renewed handlers so a new stale episode starts a fresh streak.
- Position-based paging (re-review P1): replace the currentPeriodEnd cursor with
Convex .paginate(). A currentPeriodEnd-only cursor could permanently strand
>scanLimit rows sharing one currentPeriodEnd (a tie); opaque position paging
handles ties and skips drained/backed-off pages without re-scanning.
- Move the terminal-not-found downgrade decision out of reconcileOneStaleRow into
the loop (so the breaker can gate it).
- Tests: mass-404 breaker (10 confirmed 404s -> 5 downgraded, rest active, alert
logged); a single 404 after an unrelated prior failure does NOT downgrade.
Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
… cycle Two items caught by the adversarial re-review of the mass-404 breaker round: - Guard the downgrade mutation (P2): moving the terminal-not-found downgrade out of reconcileOneStaleRow into the action loop left the expireMissingDodoSubscription runMutation unguarded, so an OCC rejection (concurrent webhook on the same sub) or a recompute error would propagate out of the loop and skip continuation scheduling — the exact isolation invariant this PR hardened. Wrap it: on throw, log + safeMarkReconcileAttempt + count failed + continue. - Per-cycle circuit breaker (P3): the downgrade count and halt latch were per-invocation locals, so a continuation chain could downgrade up to MAX_CONTINUATIONS x CAP per cycle. Thread notFoundDowngradesSoFar + a massNotFoundHalted latch through continuation args so the bound is per cron cycle (absolute cap total across the chain; a per-invocation majority latches the halt for the rest of the cycle). - Test: with limit 3 across a 12-row all-404 backlog, the whole cycle downgrades exactly 2 (majority-latched in invocation 1), not 2 per continuation. Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
da96ad5 to
eddcbfd
Compare
|
Rebased onto current
Verification on the rebased tree: 657/657 convex tests (incl. the renewal-reconciliation suite), |
Summary
Missed Dodo renewal webhooks no longer leave paying subscribers stuck behind stale local entitlement periods. A daily Convex reconciliation now finds active local subscriptions whose period has passed, checks Dodo for authoritative subscription state, refreshes the local period/status/product projection, and recomputes entitlements through the existing multi-subscription precedence helper.
The reconciliation runs in bounded batches, uses a
(status, currentPeriodEnd)subscription index instead of scanning every row, and treats per-subscription Dodo failures as isolated operational failures so one bad lookup does not block other stale subscribers. Unknown paid product IDs follow the existing webhook fail-open posture by over-grantingenterprisewhile logging for catalog repair.Validation
npm run test:convex -- convex/__tests__/billing.test.tsnpm run typechecknpm run typecheck:apigit diff --checkFixes #4765