feat(activation): persist wizard outcome + activation-lift analysis script#5584
Merged
koala73 merged 3 commits intoJul 25, 2026
Merged
Conversation
…cript Neither the Umami funnel side (no userId, joinable to nothing) nor the Convex side (no outcome record) could answer "did the Pro Activation wizard improve feature adoption" -- see #5582 for the full investigation, including the #5565 Umami outage that blacked out ~4 days spanning most of #5534's launch window. - convex/schema.ts: extend proActivationPresentations with confirmedSteps/ skippedSteps/failedSteps/exitedAt -- the durable per-subscription outcome record, independent of Umami. - convex/payments/billing.ts: new recordProActivationOutcome mutation, ownership + nonce checked like confirmProActivationPresentation. Best-effort by design (no-ops on a stale/mismatched presentation row). - src/services/billing.ts: client wrapper, swallows errors so a failed write never blocks the subscriber's exit. - src/services/pro-activation-state.ts: buildActivationOutcomeBuckets, a pure helper (alongside the existing summarizeActivationExit) that buckets step results by outcome -- unit-testable without a DOM. - src/components/ProActivationInterstitial.ts: call the new mutation from onExit, gated to the markerless cohort (the only flow with a presentation row to attach outcomes to). - scripts/report-activation-lift.mjs: read-only report comparing feature creation (notificationChannels/alertRules/userApiKeys/mcpProTokens) after wizard exit, engaged vs. presented-only-and-skipped-everything. Not a randomized control -- both groups were equally shown the wizard, so this isolates an engagement effect, not the wizard's existence. Enforces a 30-sample floor per group before printing a rate comparison. Verified against real (read-only) production Convex data: the new table/ fields don't exist yet since #5564 hasn't merged, and the script reports that state cleanly rather than erroring.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
# Conflicts: # convex/__tests__/billing.test.ts
koala73
merged commit Jul 25, 2026
075331f
into
codex/pro-activation-first-cycle-backfill
26 checks passed
koala73
added a commit
that referenced
this pull request
Jul 25, 2026
…p from one source - export proActivationStepId from convex/schema.ts and import it in convex/payments/billing.ts instead of a hand-duplicated literal union (schema.ts and billing.ts previously listed the same 3 literals independently, with nothing to catch drift) - derive MAX_PRO_ACTIVATION_OUTCOME_REVISION and the disjoint-steps bound from proActivationStepId.members.length instead of hardcoded literals (4 and 3), which had zero headroom above the wizard's exact worst-case call count and would silently start rejecting writes if a step or a progress-call site were ever added Found in ce-code-review of #5584 (forward-ported here as #5590); both findings independently validated. Convex billing suite (149/149) and full typecheck pass unchanged.
koala73
added a commit
that referenced
this pull request
Jul 25, 2026
…stitial openProActivationFlow opened the interstitial (wiring onProgress/onExit to recordProActivationOutcome) BEFORE awaiting confirmPresentationWithRetry. If a user interacted with a step in the window before that awaited call resolved and confirmPresentationWithRetry then failed (lost claim ownership, or transport errors exhausting the retry schedule), recordProActivationOutcome's own defensive presentedAt backfill had already fired -- permanently blocking a legitimate future re-claim via claimProActivationPresentation's already_presented check, even though the server never actually acknowledged this presentation. Move the confirmPresentationWithRetry await ahead of dependencies.openInterstitial(...) so the interstitial (and therefore its onProgress/onExit handlers) cannot exist until the presentation is genuinely confirmed. subscribeAuthState's reactive stale-account guard is unaffected: it still subscribes immediately after open and checks current state synchronously on subscribe. Adds a regression test (e2e/pro-activation.spec.ts) that holds confirm pending, asserts the interstitial is not mounted while it is in flight, then resolves confirm with failure and asserts no outcome write occurred. Verified red on the prior ordering (overlay count 1 while confirm was pending) and green on this fix. Found in ce-code-review of #5584 (forward-ported here as #5590; the presentedAt/claimProActivationPresentation code is unchanged by later outcomeTrackingVersion commits on this branch). Full e2e/pro-activation spec (22/22), frontend typecheck, and lint pass.
koala73
added a commit
that referenced
this pull request
Jul 25, 2026
…give-up path Two test gaps flagged in ce-code-review of #5584/#5590: - tests/report-activation-lift.test.mjs: no test drove analyzeActivationLift() to the 'no-mature-outcomes' verdict -- the state the script is realistically in immediately post-launch, before any presentation has completed its observation window. Adds a case with every presentation still immature and asserts the verdict, empty engaged/presentedOnly summaries, and the formatted report message. - e2e/pro-activation.spec.ts: no test exercised persistActivationOutcomeWithRetry's retry-then-give-up path -- the existing recordOutcome mock always resolved true. Adds an outcomeAlwaysFails harness option and a test asserting exactly 3 attempts (1 initial + the 2 scheduled OUTCOME_WRITE_RETRY_DELAYS_MS retries), that it stops retrying afterward, and that the best-effort failure never blocks the flow (summary stays interactive, finish still closes the overlay). Full e2e/pro-activation spec (23/23), report-activation-lift tests (7/7), typecheck, and lint pass.
This was referenced Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #5534/#5564. Neither the Umami funnel side (events carry no
userId, unjoinable to Convex feature-usage tables) nor the Convex side (no outcome record existed) could answer "did the Pro Activation wizard actually improve feature adoption?" — see #5582 for the full investigation, including the #5565 Umami outage that blacked out ~4 days spanning almost all of #5534's post-launch window.Base note: targets
codex/pro-activation-first-cycle-backfill(#5564), notmain— this PR extendsproActivationPresentations, which #5564 introduces. Depends on #5564 merging first; this diff shows only the incremental work.What this adds
convex/schema.ts:proActivationPresentationsgainsconfirmedSteps/skippedSteps/failedSteps/exitedAt— the durable per-subscription outcome record.convex/payments/billing.ts:recordProActivationOutcomemutation, ownership + nonce checked like the existingconfirmProActivationPresentation. Best-effort — no-ops on a stale/mismatched presentation row rather than erroring on a subscriber who's already leaving.src/services/pro-activation-state.ts:buildActivationOutcomeBuckets, a pure helper alongside the existingsummarizeActivationExit— unit-testable without a DOM.src/components/ProActivationInterstitial.ts: calls the new mutation fromonExit, gated to the markerless cohort (the only flow with a presentation row to attach outcomes to).scripts/report-activation-lift.mjs: read-only report comparing post-exit feature creation (notificationChannels/alertRules/userApiKeys/mcpProTokens) between subscribers who confirmed >=1 step vs. subscribers who were shown the wizard and confirmed nothing. Not a randomized control — both groups were equally exposed, so this isolates an engagement effect, not the wizard's existence. Enforces a 30-sample floor per group before printing a rate comparison, to avoid the exact small-sample trap that motivated this issue.Verification
tsc --noEmitandtsc --noEmit -p tsconfig.api.jsonclean.convex/__tests__/billing.test.tspass (2 new: outcome persists correctly; no-ops on nonce mismatch / wrong user / never-claimed subscription).pro-activation-controller,pro-activation-state,widget-store,widget-buildertest files (4 new:buildActivationOutcomeBucketsbucketing/ordering/empty-flow cases).scripts/report-activation-lift.mjsrun against real (read-only) production Convex data: correctly reports the table doesn't exist yet (since feat(activation): onboard unactivated first-cycle Pro users #5564 hasn't merged) instead of crashing.Non-goals (v1)
Closes #5582.