Summary

Neither the Umami funnel side nor the Convex side can currently answer "did the Pro Activation onboarding wizard (#5534, extended by #5564) improve feature adoption?" — not because the mechanism is broken, but because no durable, queryable outcome record exists.

Current state

• trackProActivation() (src/services/analytics.ts) fires entered/stepConfirmed/stepSkipped/exit events to self-hosted Umami with planKey/step/aggregate exit counts only — no userId, so it cannot be joined against Convex feature-usage data per subscriber.
• Umami was down 2026-07-20 17:05 UTC → 2026-07-24 20:24 UTC (see bug(analytics): Umami collector dead since 2026-07-20 17:05Z — OOM crash-loop, 502, ~4 days of product analytics lost (blinds #5534 funnel) #5565) — a 4-day blackout spanning almost all of feat(activation): day-0 pro activation onboarding interstitial #5534's post-launch window. It's restored now, but the historical gap is permanent.
• Convex has no table recording "did this subscription's activation flow complete, and what did they do" — proActivationPresentations (added in feat(activation): onboard unactivated first-cycle Pro users #5564) only tracks the cross-device claim/lease (claimNonce, claimedAt, presentedAt), not step-level outcome.
• A manual before/after check against notificationChannels/alertRules/userApiKeys/mcpProTokens _creationTime (done during feat(activation): onboard unactivated first-cycle Pro users #5564's review, see PR comments) found only 8 events in the ~13.6h post-feat(activation): day-0 pro activation onboarding interstitial #5534-launch window — nowhere near enough sample to say anything, and with no way to attribute which of those 8 came from someone who actually went through the wizard.

Proposal

1. Extend proActivationPresentations with outcome fields: confirmedSteps: v.optional(v.array(v.string())), skippedSteps: v.optional(v.array(v.string())), exitedAt: v.optional(v.number()). Wire ProActivationInterstitial.ts's exit handler to persist this via a new/extended Convex mutation, alongside (not instead of) the existing Umami event.
2. Analysis query/script joining proActivationPresentations rows against notificationChannels/alertRules/userApiKeys/mcpProTokens _creationTime for the same userId, filtered to createdAt > exitedAt within a configurable window (start with 14d).
3. Comparison group: subscribers with confirmedSteps.length > 0 vs. subscribers with presentedAt set but zero confirmed steps (saw the wizard, skipped everything) — both groups are equally eligible/exposed, so this isolates an engagement effect without a true control arm.
4. Start as an on-demand script (scripts/report-activation-lift.mjs or similar) rather than a live dashboard; upgrade later if useful.

Non-goals for v1

• No live dashboard.
• No true randomized control (would mean deliberately not onboarding some eligible users — not proposing that).
• Not retroactive — the 4-day Umami blackout and the pre-feat(activation): onboard unactivated first-cycle Pro users #5564 launch window have no recoverable step-level data; this only measures forward from when the instrumentation ships.

Found while

Reviewing PR #5564, when asked whether onboarding measurably improves feature adoption — answer was "no data exists to say either way yet," which surfaced this gap.