Split from the payment reliability review around #4765.

Problem

Dodo webhook processing is intentionally fail-fast for malformed subscription events so the provider retries. That is better than silently dropping a bad event, but there is still no durable local dead-letter / incident surface when retries are exhausted or a permanent schema mismatch keeps failing.

Current risk:

• A bad subscription/payment webhook can fail repeatedly and exhaust provider retries.
• The missing event can leave subscription, payment, or entitlement state stale.
• Operators have to infer the failure from logs/Sentry/provider dashboard rather than a first-class local record.

This is especially important for renewal events because a missed subscription.renewed can effectively downgrade a paying subscriber once local validUntil expires.

Proposed work

Add durable webhook failure tracking for Dodo payment/subscription events.

Suggested shape:

1. Create a paymentWebhookFailures / billingIncidents table keyed by webhookId, eventType, and, when available, subscription_id / payment_id.
2. On processing failure, persist sanitized failure metadata: error kind, message, data keys, event timestamp, receivedAt, retry count/lastSeenAt if repeated.
3. Do not store full sensitive payloads unless explicitly sanitized.
4. Emit Sentry/ops alerts when failures are new, repeated, or older than an SLA.
5. Provide an internal action/mutation to mark incidents resolved after manual repair or reconciliation.
6. Add a health/diagnostic query so on-call can see unresolved payment webhook failures.

Acceptance criteria

• Permanent webhook processing failures leave a durable local failure record rather than only logs.
• Repeated failures update the same record idempotently instead of inserting unbounded duplicates.
• Failure records include enough context to repair affected subscription/payment state without exposing secrets or raw PII-heavy payloads.
• New/repeated unresolved failures emit an ops signal with counts and affected event types.
• Tests cover malformed subscription event, repeated same webhook id, different webhook ids for same subscription, sanitized metadata, and resolve flow.

Related

• fix(payments): add reconciliation for missed Dodo renewal webhooks #4765 missed Dodo renewal webhook reconciliation
• feat(payments): P2 — daily reconciliation cron for payments stuck in requires_customer_action #4470 stuck payment-action reconciliation
• fix(payments): re-check Dodo before hard-denying recently stale active subscriptions #4770 on-demand re-check before hard denial

Priority: P2. This is an observability and repairability layer for the payment pipeline; it turns silent webhook exhaustion into an actionable queue.