Skip to content

feat(activation): persist wizard outcome + activation-lift analysis script#5584

Merged
koala73 merged 3 commits into
codex/pro-activation-first-cycle-backfillfrom
feat/activation-lift-tracking
Jul 25, 2026
Merged

feat(activation): persist wizard outcome + activation-lift analysis script#5584
koala73 merged 3 commits into
codex/pro-activation-first-cycle-backfillfrom
feat/activation-lift-tracking

Conversation

@koala73

@koala73 koala73 commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

Follow-up to #5534/#5564. Neither the Umami funnel side (events carry no userId, unjoinable to Convex feature-usage tables) nor the Convex side (no outcome record existed) could answer "did the Pro Activation wizard actually improve feature adoption?" — see #5582 for the full investigation, including the #5565 Umami outage that blacked out ~4 days spanning almost all of #5534's post-launch window.

Base note: targets codex/pro-activation-first-cycle-backfill (#5564), not main — this PR extends proActivationPresentations, which #5564 introduces. Depends on #5564 merging first; this diff shows only the incremental work.

What this adds

  • convex/schema.ts: proActivationPresentations gains confirmedSteps/skippedSteps/failedSteps/exitedAt — the durable per-subscription outcome record.
  • convex/payments/billing.ts: recordProActivationOutcome mutation, ownership + nonce checked like the existing confirmProActivationPresentation. Best-effort — no-ops on a stale/mismatched presentation row rather than erroring on a subscriber who's already leaving.
  • src/services/pro-activation-state.ts: buildActivationOutcomeBuckets, a pure helper alongside the existing summarizeActivationExit — unit-testable without a DOM.
  • src/components/ProActivationInterstitial.ts: calls the new mutation from onExit, gated to the markerless cohort (the only flow with a presentation row to attach outcomes to).
  • scripts/report-activation-lift.mjs: read-only report comparing post-exit feature creation (notificationChannels/alertRules/userApiKeys/mcpProTokens) between subscribers who confirmed >=1 step vs. subscribers who were shown the wizard and confirmed nothing. Not a randomized control — both groups were equally exposed, so this isolates an engagement effect, not the wizard's existence. Enforces a 30-sample floor per group before printing a rate comparison, to avoid the exact small-sample trap that motivated this issue.

Verification

  • tsc --noEmit and tsc --noEmit -p tsconfig.api.json clean.
  • Convex: 142/142 convex/__tests__/billing.test.ts pass (2 new: outcome persists correctly; no-ops on nonce mismatch / wrong user / never-claimed subscription).
  • Frontend: 343/343 across pro-activation-controller, pro-activation-state, widget-store, widget-builder test files (4 new: buildActivationOutcomeBuckets bucketing/ordering/empty-flow cases).
  • scripts/report-activation-lift.mjs run against real (read-only) production Convex data: correctly reports the table doesn't exist yet (since feat(activation): onboard unactivated first-cycle Pro users #5564 hasn't merged) instead of crashing.

Non-goals (v1)

  • No live dashboard — on-demand script only.
  • No true randomized control.
  • Not retroactive — the Umami blackout and pre-this-PR launch window have no recoverable step-level data.

Closes #5582.

…cript

Neither the Umami funnel side (no userId, joinable to nothing) nor the
Convex side (no outcome record) could answer "did the Pro Activation
wizard improve feature adoption" -- see #5582 for the full investigation,
including the #5565 Umami outage that blacked out ~4 days spanning most
of #5534's launch window.

- convex/schema.ts: extend proActivationPresentations with confirmedSteps/
  skippedSteps/failedSteps/exitedAt -- the durable per-subscription outcome
  record, independent of Umami.
- convex/payments/billing.ts: new recordProActivationOutcome mutation,
  ownership + nonce checked like confirmProActivationPresentation.
  Best-effort by design (no-ops on a stale/mismatched presentation row).
- src/services/billing.ts: client wrapper, swallows errors so a failed
  write never blocks the subscriber's exit.
- src/services/pro-activation-state.ts: buildActivationOutcomeBuckets, a
  pure helper (alongside the existing summarizeActivationExit) that buckets
  step results by outcome -- unit-testable without a DOM.
- src/components/ProActivationInterstitial.ts: call the new mutation from
  onExit, gated to the markerless cohort (the only flow with a presentation
  row to attach outcomes to).
- scripts/report-activation-lift.mjs: read-only report comparing feature
  creation (notificationChannels/alertRules/userApiKeys/mcpProTokens) after
  wizard exit, engaged vs. presented-only-and-skipped-everything. Not a
  randomized control -- both groups were equally shown the wizard, so this
  isolates an engagement effect, not the wizard's existence. Enforces a
  30-sample floor per group before printing a rate comparison.

Verified against real (read-only) production Convex data: the new table/
fields don't exist yet since #5564 hasn't merged, and the script reports
that state cleanly rather than erroring.
@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment Jul 25, 2026 6:36am

Request Review

@koala73
koala73 merged commit 075331f into codex/pro-activation-first-cycle-backfill Jul 25, 2026
26 checks passed
koala73 added a commit that referenced this pull request Jul 25, 2026
…p from one source

- export proActivationStepId from convex/schema.ts and import it in
  convex/payments/billing.ts instead of a hand-duplicated literal union
  (schema.ts and billing.ts previously listed the same 3 literals
  independently, with nothing to catch drift)
- derive MAX_PRO_ACTIVATION_OUTCOME_REVISION and the disjoint-steps bound
  from proActivationStepId.members.length instead of hardcoded literals
  (4 and 3), which had zero headroom above the wizard's exact worst-case
  call count and would silently start rejecting writes if a step or a
  progress-call site were ever added

Found in ce-code-review of #5584 (forward-ported here as #5590); both
findings independently validated. Convex billing suite (149/149) and
full typecheck pass unchanged.
koala73 added a commit that referenced this pull request Jul 25, 2026
…stitial

openProActivationFlow opened the interstitial (wiring onProgress/onExit to
recordProActivationOutcome) BEFORE awaiting confirmPresentationWithRetry.
If a user interacted with a step in the window before that awaited call
resolved and confirmPresentationWithRetry then failed (lost claim
ownership, or transport errors exhausting the retry schedule),
recordProActivationOutcome's own defensive presentedAt backfill had
already fired -- permanently blocking a legitimate future re-claim via
claimProActivationPresentation's already_presented check, even though the
server never actually acknowledged this presentation.

Move the confirmPresentationWithRetry await ahead of
dependencies.openInterstitial(...) so the interstitial (and therefore its
onProgress/onExit handlers) cannot exist until the presentation is
genuinely confirmed. subscribeAuthState's reactive stale-account guard is
unaffected: it still subscribes immediately after open and checks current
state synchronously on subscribe.

Adds a regression test (e2e/pro-activation.spec.ts) that holds confirm
pending, asserts the interstitial is not mounted while it is in flight,
then resolves confirm with failure and asserts no outcome write occurred.
Verified red on the prior ordering (overlay count 1 while confirm was
pending) and green on this fix.

Found in ce-code-review of #5584 (forward-ported here as #5590; the
presentedAt/claimProActivationPresentation code is unchanged by later
outcomeTrackingVersion commits on this branch). Full e2e/pro-activation
spec (22/22), frontend typecheck, and lint pass.
koala73 added a commit that referenced this pull request Jul 25, 2026
…give-up path

Two test gaps flagged in ce-code-review of #5584/#5590:

- tests/report-activation-lift.test.mjs: no test drove
  analyzeActivationLift() to the 'no-mature-outcomes' verdict -- the state
  the script is realistically in immediately post-launch, before any
  presentation has completed its observation window. Adds a case with
  every presentation still immature and asserts the verdict, empty
  engaged/presentedOnly summaries, and the formatted report message.

- e2e/pro-activation.spec.ts: no test exercised
  persistActivationOutcomeWithRetry's retry-then-give-up path -- the
  existing recordOutcome mock always resolved true. Adds an
  outcomeAlwaysFails harness option and a test asserting exactly 3 attempts
  (1 initial + the 2 scheduled OUTCOME_WRITE_RETRY_DELAYS_MS retries), that
  it stops retrying afterward, and that the best-effort failure never
  blocks the flow (summary stays interactive, finish still closes the
  overlay).

Full e2e/pro-activation spec (23/23), report-activation-lift tests (7/7),
typecheck, and lint pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant