Skip to content

fix(payments): reconcile missed Dodo renewals#4794

Merged
koala73 merged 6 commits into
mainfrom
codex/issue-4765-dodo-renewal-reconcile
Jul 7, 2026
Merged

fix(payments): reconcile missed Dodo renewals#4794
koala73 merged 6 commits into
mainfrom
codex/issue-4765-dodo-renewal-reconcile

Conversation

@koala73

@koala73 koala73 commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Summary

Missed Dodo renewal webhooks no longer leave paying subscribers stuck behind stale local entitlement periods. A daily Convex reconciliation now finds active local subscriptions whose period has passed, checks Dodo for authoritative subscription state, refreshes the local period/status/product projection, and recomputes entitlements through the existing multi-subscription precedence helper.

The reconciliation runs in bounded batches, uses a (status, currentPeriodEnd) subscription index instead of scanning every row, and treats per-subscription Dodo failures as isolated operational failures so one bad lookup does not block other stale subscribers. Unknown paid product IDs follow the existing webhook fail-open posture by over-granting enterprise while logging for catalog repair.

Validation

  • npm run test:convex -- convex/__tests__/billing.test.ts
  • npm run typecheck
  • npm run typecheck:api
  • git diff --check
  • pre-push hook: typecheck, API typecheck, Convex type check, CJS syntax, Unicode safety, boundary lint, safe HTML, Sentry coverage, rate-limit policy coverage, premium-fetch parity, version sync

Fixes #4765


Compound Engineering
GPT--5

@vercel

vercel Bot commented Jul 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment Jul 6, 2026 7:10pm

Request Review

@greptile-apps

greptile-apps Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a daily Convex reconciliation job that finds paying subscribers with locally stale subscription periods, queries Dodo's API for authoritative state, and refreshes the local subscription + entitlement records through the existing multi-subscription precedence helper.

  • A new compound by_status_period_end index enables efficient O(stale) queries instead of full-table scans, and the reconciliation runs in bounded batches of 50 with per-subscription error isolation.
  • A new reconcileMissedDodoRenewals internal action (called daily via cron) drives a listStaleActiveSubscriptionsForRenewalReconciliation query and an applyDodoSubscriptionReconciliation mutation, with proper idempotency guards inside the mutation to handle concurrent webhook delivery.

Confidence Score: 4/5

Safe to merge for normal operating conditions; the two gaps only matter during a large Dodo webhook outage and do not affect correctness for the common case.

The reconciliation logic itself — idempotency guards in the mutation, error isolation in the action loop, the compound index query, and entitlement recompute via the existing helper — all look correct. The two concerns are edge-case design gaps: no second-pass scheduling when hasMore is true means a large backlog clears at 50/day, and subscriptions with unrecognized Dodo statuses sit permanently in the stale query results burning a batch slot per day.

convex/crons.ts and the unsupported-status skip branch in convex/payments/billing.ts deserve a second look if large-scale webhook outages are a plausible operational scenario.

Important Files Changed

Filename Overview
convex/payments/billing.ts Core reconciliation logic: adds listStaleActiveSubscriptionsForRenewalReconciliation, applyDodoSubscriptionReconciliation, and reconcileMissedDodoRenewals. Logic and guard conditions look correct; two design-level concerns around batch backlog handling and unsupported-status subscriptions.
convex/schema.ts Adds by_status_period_end compound index on [status, currentPeriodEnd] — correctly structured for the equality-then-range pattern used in the reconciliation query.
convex/crons.ts Registers the daily reconciliation cron at 3:17 UTC with empty args {}, using all defaults (limit=50). No follow-up scheduling when hasMore is true.
convex/tests/billing.test.ts Adds two reconciliation tests: normal renewal application and failure isolation across subscriptions. Core happy-path and error-isolation paths are covered.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Cron as Daily Cron (3:17 UTC)
    participant Action as reconcileMissedDodoRenewals
    participant Query as listStaleActiveSubscriptions
    participant Dodo as Dodo API
    participant Mutation as applyDodoSubscriptionReconciliation
    participant DB as Convex DB

    Cron->>Action: "run({})"
    Action->>Query: "runQuery({now, limit=50})"
    Query->>DB: subscriptions.by_status_period_end(active, lt now).take(50)
    DB-->>Query: stale[]
    Query-->>Action: stale subscriptions

    loop for each stale subscription
        Action->>Dodo: subscriptions.retrieve(dodoSubscriptionId)
        alt Dodo call succeeds
            Dodo-->>Action: remote subscription
            Action->>Action: normalizeRemoteSubscription(remote)
            alt unsupported status
                Action->>Action: skipped++, warn log
            else supported status
                Action->>Mutation: "runMutation({subscriptionId, remote, observedAt})"
                Mutation->>DB: db.get(subscriptionId)
                alt local no longer stale
                    Mutation-->>Action: skipped / local_no_longer_stale
                else remote not newer
                    Mutation-->>Action: skipped / remote_not_newer
                else needs update
                    Mutation->>DB: db.patch(subscription)
                    Mutation->>DB: recomputeEntitlementFromAllSubs(userId)
                    Mutation-->>Action: reconciled
                end
            end
        else Dodo call fails
            Action->>Action: failed++, error log (continue loop)
        end
    end

    Action-->>Cron: "ReconciliationSummary{inspected, reconciled, skipped, failed, hasMore}"
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Cron as Daily Cron (3:17 UTC)
    participant Action as reconcileMissedDodoRenewals
    participant Query as listStaleActiveSubscriptions
    participant Dodo as Dodo API
    participant Mutation as applyDodoSubscriptionReconciliation
    participant DB as Convex DB

    Cron->>Action: "run({})"
    Action->>Query: "runQuery({now, limit=50})"
    Query->>DB: subscriptions.by_status_period_end(active, lt now).take(50)
    DB-->>Query: stale[]
    Query-->>Action: stale subscriptions

    loop for each stale subscription
        Action->>Dodo: subscriptions.retrieve(dodoSubscriptionId)
        alt Dodo call succeeds
            Dodo-->>Action: remote subscription
            Action->>Action: normalizeRemoteSubscription(remote)
            alt unsupported status
                Action->>Action: skipped++, warn log
            else supported status
                Action->>Mutation: "runMutation({subscriptionId, remote, observedAt})"
                Mutation->>DB: db.get(subscriptionId)
                alt local no longer stale
                    Mutation-->>Action: skipped / local_no_longer_stale
                else remote not newer
                    Mutation-->>Action: skipped / remote_not_newer
                else needs update
                    Mutation->>DB: db.patch(subscription)
                    Mutation->>DB: recomputeEntitlementFromAllSubs(userId)
                    Mutation-->>Action: reconciled
                end
            end
        else Dodo call fails
            Action->>Action: failed++, error log (continue loop)
        end
    end

    Action-->>Cron: "ReconciliationSummary{inspected, reconciled, skipped, failed, hasMore}"
Loading

Reviews (1): Last reviewed commit: "fix(payments): reconcile missed Dodo ren..." | Re-trigger Greptile

Comment thread convex/crons.ts
Comment on lines 76 to +82
);

crons.daily(
"dodo-renewal-reconciliation",
{ hourUTC: 3, minuteUTC: 17 },
internal.payments.billing.reconcileMissedDodoRenewals,
{},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No follow-up scheduling when hasMore: true

The cron always fires with {}, which uses the hard cap of 50. When stale.length === limit the action returns hasMore: true, but there is no mechanism to schedule a second pass within the same day. During a Dodo webhook outage that lets hundreds of subscriptions go stale, only the 50 oldest are reconciled each calendar day — users beyond position 50 whose subscriptions were actually cancelled or expired in Dodo will continue to receive entitlements they are no longer entitled to until their slot is reached on a future day's run.

Comment thread convex/payments/billing.ts Outdated
Comment on lines +604 to +611

const normalized = normalizeRemoteSubscription(remote);
if (normalized.kind === "unsupported-status") {
summary.skipped++;
console.warn(
`[billing/reconcile] Skipping subscription ${normalized.dodoSubscriptionId}: unsupported Dodo status "${normalized.status}"`,
);
continue;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unsupported-status subscriptions permanently hold batch slots

When Dodo returns a status value outside active | on_hold | cancelled | expired (e.g., if Dodo introduces a new lifecycle state), normalizeRemoteSubscription returns kind: "unsupported-status", the action increments skipped, and the local record is left untouched — status = "active", currentPeriodEnd still in the past. Because the record is never updated, it will reappear in every subsequent daily cron run, consuming one of the 50 available batch slots and generating a Dodo API call, indefinitely. A subscriber stuck in this state also retains their entitlement forever since isCoveringAt treats status === "active" as always-covering regardless of period end.

koala73 added a commit that referenced this pull request Jul 5, 2026
…round 2)

Fold in the reliability + test findings from the ce-code-review pass:

- Bound the Dodo poll (`payments.retrieve`) with `{ timeout: 10s,
  maxRetries: 1 }`. The SDK default is 60s x 2 retries (~180s); on a Dodo
  outage one degraded poll could otherwise stall the whole sequential
  batch — the exact failure this cron exists to survive. Mirrors the
  Resend timeout already added.
- Run the reconciliation cron every 6h instead of daily. A payment
  becomes a candidate at ~6h pending, so on a daily cadence its age at
  first scan is uniformly 6h-30h and anything in (24h, 30h] misses the
  24h customer-email freshness gate — ~25% of stuck payments silently
  dropped to ops-only. At 6h cadence first-scan age stays <=~12h. Safe:
  the action is idempotent and marker-gated.
- Dropped-webhook guard: when a reconciled SUCCEEDED payment carries a
  subscription id but has no subscriptions row, console.error (Sentry).
  A dropped payment webhook may travel with a dropped subscription.active
  webhook — charged customer, no entitlement, and #4794's renewal
  reconciler only scans active rows, so nobody else catches it.
- Correct the scan-cap comment/message: newest-first means over-cap OLD
  rows are deferred indefinitely (until the backlog clears), not "to a
  later run".
- Tests: action-level unrecognised-status (requires_payment_method) path,
  two-run at-most-once-email idempotency, and the dropped-subscription
  guard (pages / does not page).

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
koala73 added a commit that referenced this pull request Jul 6, 2026
* feat(payments): reconcile stuck pending payments

* fix(payments): harden stuck-pending reconciliation (PR #4795 review)

Address the adversarial review of the stuck-pending payment cron:

- F1: collapse unrecognised non-terminal Dodo IntentStatus (incl. null /
  requires_payment_method — the typical abandoned-3DS end-state) to
  still-pending, writing a marker that records the raw observed status in
  every fall-through. Previously these returned unknown_status with NO
  marker → daily re-poll for 14 days + batch-slot starvation.
- F2: page ops for real. The still-pending ops branch now console.errors
  (Convex auto-Sentry, refund-alert precedent) instead of a never-surfaced
  console.warn.
- F3: claim the reconciliation marker BEFORE sending the customer email
  (idempotency barrier — a post-send failure can't re-email next day),
  finalize after; split failure reporting by phase (poll / email / record).
- F4: gate the "continue checkout" email on link freshness (<24h). Older
  candidates route to ops rather than emailing a dead Dodo checkout link.
- F5: scan newest-first (.order desc) so freshly-stuck rows don't fall off
  the 500-row scan cap; log when the cap is hit.
- F6: guard ctx.scheduler.runAfter with its own try/catch; add
  AbortSignal.timeout to the Resend POST; stop logging the raw Resend
  response body (can echo the recipient email).
- F7: add reconciliation tests (unknown/null status marker, marker-before-
  email ordering, already_terminal race, Resend-throw-mid-batch isolation,
  scan-cap ordering).

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ

* fix(payments): reconciliation reliability hardening (PR #4795 review round 2)

Fold in the reliability + test findings from the ce-code-review pass:

- Bound the Dodo poll (`payments.retrieve`) with `{ timeout: 10s,
  maxRetries: 1 }`. The SDK default is 60s x 2 retries (~180s); on a Dodo
  outage one degraded poll could otherwise stall the whole sequential
  batch — the exact failure this cron exists to survive. Mirrors the
  Resend timeout already added.
- Run the reconciliation cron every 6h instead of daily. A payment
  becomes a candidate at ~6h pending, so on a daily cadence its age at
  first scan is uniformly 6h-30h and anything in (24h, 30h] misses the
  24h customer-email freshness gate — ~25% of stuck payments silently
  dropped to ops-only. At 6h cadence first-scan age stays <=~12h. Safe:
  the action is idempotent and marker-gated.
- Dropped-webhook guard: when a reconciled SUCCEEDED payment carries a
  subscription id but has no subscriptions row, console.error (Sentry).
  A dropped payment webhook may travel with a dropped subscription.active
  webhook — charged customer, no entitlement, and #4794's renewal
  reconciler only scans active rows, so nobody else catches it.
- Correct the scan-cap comment/message: newest-first means over-cap OLD
  rows are deferred indefinitely (until the backlog clears), not "to a
  later run".
- Tests: action-level unrecognised-status (requires_payment_method) path,
  two-run at-most-once-email idempotency, and the dropped-subscription
  guard (pages / does not page).

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
koala73 added 6 commits July 6, 2026 23:00
…e statuses, ordering)

Address adversarial review of the missed-Dodo-renewal reconciler:

- Poison-row starvation: record per-row reconcile attempts
  (lastReconcileAttemptAt + reconcileFailureCount) with exponential backoff
  so a permanently-failing row (e.g. test-mode-era sub 404ing the live
  client) stops re-occupying the ordered scan window every run. Widen the
  candidate scan, filter backed-off rows, and continuation-schedule the
  remainder within the cron cycle (bounded by MAX_CONTINUATIONS).
- Remote failed/pending: map Dodo `failed` -> local `expired` (entitlement
  recompute downgrades instead of granting forever via isCoveringAt); skip
  `pending`/unknown explicitly and escalate the skip log to console.error so
  Convex auto-Sentry captures it.
- Ordering guard: add the webhook handlers' isNewerEvent(existing.updatedAt,
  observedAt) check so a concurrent subscription.plan_changed write is not
  clobbered by the cron's stale read.
- Runtime cap: bail the loop on a wall-clock budget well under Convex's
  10-min action cap and continuation-schedule the rest.
- DRY: reuse the exported resolvePlanKey webhook resolver instead of the
  duplicated productPlans -> alias -> enterprise-fallback chain.
- Tests: enterprise fallback, remote-cancelled downgrade, remote-failed ->
  expired, both skip guards, ordering guard, unsupported/pending skips, and
  the starvation/attempt-marker + continuation drain behavior.

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
…empt marks

Address ce-code-review findings (performance P2×2/P3, reliability P1,
testing P1/P2, adversarial P2, maintainability P2) on the reconciler:

- Saturation stall (P1, cross-model + perf): the fixed 500-row scan filtered
  backoff eligibility in memory, so a window fully within its cooldown left
  eligible=[] with attempted=0 and scheduled no continuation — silently
  stranding healthy stale rows behind it. Thread a currentPeriodEnd cursor
  through continuations: advance past a drained/saturated window (guaranteeing
  forward progress even when attempted=0) and emit a distinguishable
  console.error when a window is cooldown-saturated. Adds a clamped `scanLimit`
  arg (ops knob + deterministic test seam).
- Single-transaction backoff (perf P2): fold the attempt bookkeeping into
  applyDodoSubscriptionReconciliation's still-stale skip branches instead of a
  second markDodoReconcileAttempt round-trip; reorder guards so the concurrent
  skips are known stale-active. markDodoReconcileAttempt now only serves the
  paths that never reach apply (Dodo lookup failure, unsupported/pending).
- Safe attempt marking (reliability P1): the bookkeeping mutation now swallows
  and logs its own errors so an OCC conflict can never abort the batch or skip
  continuation scheduling.
- Faster first-failure retry (adversarial P2): first failure backs off only a
  few hours (skips the rest of the cycle but retries at the next daily run) so
  a transient Dodo error does not over-delay a legitimate downgrade; repeated
  failures still back off exponentially.
- Stricter ReconciliationSkipReason union; corrected continuation-ceiling
  comment; documented residuals.
- Tests: time-budget bail (Date.now spy), first-failure-then-next-day backoff,
  cursor-advance past a saturated window, and a load-bearing ordering-guard
  assertion (seeded productPlans so a removed guard would fail it).

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
…e row

Consolidate the remaining ce-code-review findings on the renewal reconciler:

- Reliability P1 (no-SDK-retry): getDodoClient now uses maxRetries: 0. The SDK
  retry path honors a server Retry-After verbatim and could sleep minutes,
  blowing Convex's 10-min action cap; row-level backoff + continuation is the
  retry mechanism, so each lookup stays bounded by the 10s timeout.
- Reliability P1 (safe backoff write): the best-effort attempt marking is now
  a swallow-and-log helper (safeMarkReconcileAttempt), so an OCC conflict on
  the bookkeeping mutation can never propagate out of the loop and abort the
  batch or skip continuation scheduling.
- Correctness/adversarial (404 -> expired): distinguish a definitive Dodo
  NotFound (status 404) from transient errors. After a confirmation (the row
  already has >= 1 recorded failure, i.e. a 2nd consecutive 404) the row is
  downgraded to expired via expireMissingDodoSubscription + recompute, so a
  permanently-gone poison row LEAVES the active set — resolving the scan-slot
  occupation and the entitlement over-grant together. A single flaky 404 or any
  transient 5xx/network/timeout stays on the backoff path (never downgrades).
- Correctness P3: a reconcile that leaves the row still stale (remote active,
  period end still past) records a backoff instead of clearing bookkeeping,
  avoiding a redundant re-lookup next cycle.
- Maintainability P2: extract the per-row fetch->normalize->apply->interpret
  body into reconcileOneStaleRow; dedupe the status union (export
  SubscriptionStatus from subscriptionHelpers).
- P3: one-line comment acknowledging the intentional no-run-lock (isNewerEvent
  + OCC self-heal overlaps).
- Tests: 404-twice -> expired + single-404 stays active; 5xx never downgrades;
  safeMarkReconcileAttempt swallows a throwing mutation; failureCount>=2 (2-day)
  backoff eligibility.

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
… paging

Add the required mass-404 circuit breaker and fix two P1s my own adversarial
re-review caught in the prior fix commit:

- Mass-404 circuit breaker (required): cap confirmed-not-found downgrades per
  invocation at min(5, ceil(plannedAttempts/2)). Beyond it, stop downgrading,
  console.error("mass Dodo 404s … possible wrong-environment/API-key misconfig;
  further downgrades halted this run"), and route the rest to backoff. A genuine
  handful of deleted subs still gets cleaned; a config error that 404s the whole
  base downgrades at most the threshold before self-halting (and pages ops).
- Consecutive-404 confirmation (re-review P1): the terminal-404 gate used the
  generic reconcileFailureCount, so a prior unrelated failure (5xx/timeout) plus
  a single 404 could downgrade a live customer. Add a dedicated
  reconcileNotFoundCount that only advances on a definitive 404 and resets on any
  non-404 outcome, and clear all reconcile bookkeeping in the webhook
  active/renewed handlers so a new stale episode starts a fresh streak.
- Position-based paging (re-review P1): replace the currentPeriodEnd cursor with
  Convex .paginate(). A currentPeriodEnd-only cursor could permanently strand
  >scanLimit rows sharing one currentPeriodEnd (a tie); opaque position paging
  handles ties and skips drained/backed-off pages without re-scanning.
- Move the terminal-not-found downgrade decision out of reconcileOneStaleRow into
  the loop (so the breaker can gate it).
- Tests: mass-404 breaker (10 confirmed 404s -> 5 downgraded, rest active, alert
  logged); a single 404 after an unrelated prior failure does NOT downgrade.

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
… cycle

Two items caught by the adversarial re-review of the mass-404 breaker round:

- Guard the downgrade mutation (P2): moving the terminal-not-found downgrade out
  of reconcileOneStaleRow into the action loop left the expireMissingDodoSubscription
  runMutation unguarded, so an OCC rejection (concurrent webhook on the same sub)
  or a recompute error would propagate out of the loop and skip continuation
  scheduling — the exact isolation invariant this PR hardened. Wrap it: on throw,
  log + safeMarkReconcileAttempt + count failed + continue.
- Per-cycle circuit breaker (P3): the downgrade count and halt latch were
  per-invocation locals, so a continuation chain could downgrade up to
  MAX_CONTINUATIONS x CAP per cycle. Thread notFoundDowngradesSoFar + a
  massNotFoundHalted latch through continuation args so the bound is per cron
  cycle (absolute cap total across the chain; a per-invocation majority latches
  the halt for the rest of the cycle).
- Test: with limit 3 across a 12-row all-404 backlog, the whole cycle downgrades
  exactly 2 (majority-latched in invocation 1), not 2 per continuation.

Claude-Session: https://claude.ai/code/session_018FBmQG921WC4s3qLmuEKoQ
@koala73
koala73 force-pushed the codex/issue-4765-dodo-renewal-reconcile branch from da96ad5 to eddcbfd Compare July 6, 2026 19:06
@koala73

koala73 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto current main (which now has #4935 dunning/winback + #4795 stuck-pending reconciler). Conflicts resolved:

Verification on the rebased tree: 657/657 convex tests (incl. the renewal-reconciliation suite), tsc -p convex/tsconfig.json clean, typecheck:api clean, docs-stats clean.

@koala73
koala73 merged commit 240baa0 into main Jul 7, 2026
25 checks passed
@koala73
koala73 deleted the codex/issue-4765-dodo-renewal-reconcile branch July 7, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(payments): add reconciliation for missed Dodo renewal webhooks

1 participant