Skip to content

fix(activation): land outcome tracking on main#5590

Merged
koala73 merged 6 commits into
mainfrom
codex/issue-5582-forward-port
Jul 25, 2026
Merged

fix(activation): land outcome tracking on main#5590
koala73 merged 6 commits into
mainfrom
codex/issue-5582-forward-port

Conversation

@koala73

@koala73 koala73 commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

Intent

#5584 was merged into codex/pro-activation-first-cycle-backfill after #5564 had already merged into main. Its implementation and Closes #5582 reference therefore never reached the default branch.

This PR applies that implementation to current main:

  • durable confirmed/skipped/failed activation outcome snapshots in Convex
  • authenticated, nonce-bound, monotonic outcome writes
  • interstitial progress and final-exit persistence alongside existing Umami telemetry
  • a bounded, read-only activation-lift report with mature-window and minimum-sample safeguards
  • a mixed-deploy-safe outcome tracking version so legacy clients remain valid while pre-instrumentation rows stay outside the comparison cohort

Non-goals remain unchanged: no live dashboard, randomized control, retroactive reconstruction, or unrelated refactor.

Closes #5582.

Validation Matrix

Gate Result
#5584 implementation parity PASS — original patch applied with identical patch ID 1734c5465fec98e30743b12840a7e6016afeb250; follow-up changes preserve that implementation and add one shared-constants file
Final diff scope PASS — 11 files, 1,174 additions, 8 deletions
Activation state + report tests PASS — 122/122
Controller/widget source-affinity tests PASS — 228/228
Convex billing tests PASS — 149/149
Pro Activation Playwright spec PASS — 21/21
Frontend typecheck PASS
API/Convex typecheck and string-call audit PASS
Biome + safe HTML PASS — only unrelated pre-existing warnings
Boundary and staged Unicode guards PASS
Full pre-push gate PASS
GitHub/Vercel checks PASS — 24 succeeded, 1 intentionally skipped, 0 failed or pending
Read-only production report smoke PASS — 9 existing presentations exported; all 9 correctly excluded as pre-instrumentation

Review Gates

Documentation

Not applicable. The on-demand script includes its usage, read-only behavior, cohort caveat, and configuration flags; no public API or user documentation contract changes.

Screenshots / UI Evidence

Not applicable. This forward-port adds persistence and an operator report without changing rendered UI.

Residual Findings

None.

@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Jul 25, 2026 1:58pm

Request Review

…p from one source

- export proActivationStepId from convex/schema.ts and import it in
  convex/payments/billing.ts instead of a hand-duplicated literal union
  (schema.ts and billing.ts previously listed the same 3 literals
  independently, with nothing to catch drift)
- derive MAX_PRO_ACTIVATION_OUTCOME_REVISION and the disjoint-steps bound
  from proActivationStepId.members.length instead of hardcoded literals
  (4 and 3), which had zero headroom above the wizard's exact worst-case
  call count and would silently start rejecting writes if a step or a
  progress-call site were ever added

Found in ce-code-review of #5584 (forward-ported here as #5590); both
findings independently validated. Convex billing suite (149/149) and
full typecheck pass unchanged.
…stitial

openProActivationFlow opened the interstitial (wiring onProgress/onExit to
recordProActivationOutcome) BEFORE awaiting confirmPresentationWithRetry.
If a user interacted with a step in the window before that awaited call
resolved and confirmPresentationWithRetry then failed (lost claim
ownership, or transport errors exhausting the retry schedule),
recordProActivationOutcome's own defensive presentedAt backfill had
already fired -- permanently blocking a legitimate future re-claim via
claimProActivationPresentation's already_presented check, even though the
server never actually acknowledged this presentation.

Move the confirmPresentationWithRetry await ahead of
dependencies.openInterstitial(...) so the interstitial (and therefore its
onProgress/onExit handlers) cannot exist until the presentation is
genuinely confirmed. subscribeAuthState's reactive stale-account guard is
unaffected: it still subscribes immediately after open and checks current
state synchronously on subscribe.

Adds a regression test (e2e/pro-activation.spec.ts) that holds confirm
pending, asserts the interstitial is not mounted while it is in flight,
then resolves confirm with failure and asserts no outcome write occurred.
Verified red on the prior ordering (overlay count 1 while confirm was
pending) and green on this fix.

Found in ce-code-review of #5584 (forward-ported here as #5590; the
presentedAt/claimProActivationPresentation code is unchanged by later
outcomeTrackingVersion commits on this branch). Full e2e/pro-activation
spec (22/22), frontend typecheck, and lint pass.
…give-up path

Two test gaps flagged in ce-code-review of #5584/#5590:

- tests/report-activation-lift.test.mjs: no test drove
  analyzeActivationLift() to the 'no-mature-outcomes' verdict -- the state
  the script is realistically in immediately post-launch, before any
  presentation has completed its observation window. Adds a case with
  every presentation still immature and asserts the verdict, empty
  engaged/presentedOnly summaries, and the formatted report message.

- e2e/pro-activation.spec.ts: no test exercised
  persistActivationOutcomeWithRetry's retry-then-give-up path -- the
  existing recordOutcome mock always resolved true. Adds an
  outcomeAlwaysFails harness option and a test asserting exactly 3 attempts
  (1 initial + the 2 scheduled OUTCOME_WRITE_RETRY_DELAYS_MS retries), that
  it stops retrying afterward, and that the best-effort failure never
  blocks the flow (summary stays interactive, finish still closes the
  overlay).

Full e2e/pro-activation spec (23/23), report-activation-lift tests (7/7),
typecheck, and lint pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(activation): persist wizard outcome + build activation-lift analysis (follow-up to #5534/#5564)

1 participant