Skip to content

Commit 24f7b77

Browse files
sahrizviclaude
andcommitted
fix(onboarding): add launch correlation id + full-review fixes
A whole-feature review found the events individually correct but the funnel not reconstructable. These are the fixes. Correlation (see #1048): - Every event now carries `launch_id`, a random value generated once per process and shared across threads through the environment. The funnel spans the TUI main thread and the server worker, and most of it runs before any chat session exists — so the first half of the funnel stamped an empty session and the second half a real one, leaving no key to join a single run on. It is not persisted, not derived from the machine or the user, and not reused across launches. Placed in the event envelope alongside `machine_id`, so it applies to all telemetry rather than only the onboarding events. - `~/.altimate/machine-id` is now created exclusively. Both threads initialise their own copy of the telemetry module, and on a genuinely new install both can find the file missing at the same moment; with a plain write the loser overwrote the winner while each kept its own value in memory, so one first run reported two machine ids. The fallback identity broke on exactly the run it mattered most for. `wx` makes one writer fail and re-read. Abandonment: - `last_stage` no longer advertises values it can never report. `scan_gate` and `activation` both occur after `onboarding_completed`, which sets `completed` and suppresses abandonment entirely — abandonment is by definition quitting before connecting. They are removed rather than documented as live. - Added `provider_setup`, advanced by any provider choice. A user who picked Anthropic and quit during key entry was previously reported as abandoning at `model_picker`, as though they had never chosen anything. Shutdown: - `shutdown({timeoutMs})` no longer ignores the budget when a shutdown is already in flight. It returned the existing promise, which may have been running on the default 10s, so the exit path waited past its 2s budget and the worker was terminated mid-flush anyway. The in-flight flush cannot be shortened, but the caller no longer waits on it beyond its own deadline. Known and unchanged: choosing "Search all providers…" and then picking from the full catalogue records only `provider=search_all`. Fixing that means emitting from the full model picker, which is used outside first-run too, so it needs a product call. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
1 parent de8c30f commit 24f7b77

3 files changed

Lines changed: 88 additions & 9 deletions

File tree

‎docs/docs/reference/telemetry.md‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,32 @@ We collect the following categories of events:
4747
| `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
4848
| `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
4949
| `validator_retries_exhausted` | A session terminated with unresolved validator failures after exhausting the synthetic-retry budget — names of the failing validators (no failure body content). |
50-
51-
Each event includes a timestamp, anonymous session ID, CLI version, and an anonymous machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and never tied to any personal information).
50+
| `onboarding_started` | The first-run setup gate opened (fresh launch with no usable model). |
51+
| `model_picker_shown` | The provider picker was displayed. `trigger` distinguishes the first run from `/connect`, from declining Big Pickle, and from the prompt gate. |
52+
| `provider_selected` | A provider row was chosen (`altimate_gateway`, `anthropic`, `openai`, `google`, `big_pickle`, `search_all`). Recorded at the moment of choice, so a sign-in that is then cancelled still counts. |
53+
| `big_pickle_confirm_shown` / `big_pickle_choice` | The Big Pickle interstitial was shown, and what the user decided (`accept`/`cancel`). |
54+
| `gateway_device_code_issued` | The Altimate Gateway authorize URL was built and the browser open attempted. **Name note:** the flow is a browser loopback OAuth — there is no device code. The name follows the original event spec. |
55+
| `gateway_auth_completed` / `gateway_auth_failed` | Gateway sign-in outcome. `reason` is `timeout`, `denied`, or `error` — never the underlying message, which can contain the instance name. An unrecognised callback state does not reject the pending attempt, so a CSRF mismatch surfaces as `timeout`. |
56+
| `instance_connected` | Credentials received and saved. `time_to_connect_ms` runs from the start of the authorize call, so it includes the browser launch. No instance or tenant name is sent. |
57+
| `onboarding_completed` | A model is ready and chat is live. |
58+
| `scan_gate_shown` / `scan_gate_choice` | The "scan your environment?" gate appeared, and whether the user chose `scan` or `skip`. |
59+
| `environment_scan_completed` | A `project_scan` finished — `has_dbt`, `has_warehouse`, `is_repo`, `connections_found`, and a bounded list of short `degraded` detection keys. No paths, hostnames, or connection details. Fires for every scan, including `/discover`, not just onboarding. |
60+
| `sample_setup_completed` | The sample dbt project was materialised. `success`, `models`, `tables`, and `reused` — the tool is deliberately re-callable, so this is per invocation. The target path is never sent. |
61+
| `activation_menu_shown` | The activation menu was (very likely) rendered. `variant` is `warehouse` or `no_data`. **Derived** — see the note below. |
62+
| `activation_job_selected` / `first_job_completed` | Which activation job the user started and, where observable, finished. **Derived** — see the note below. |
63+
| `first_prompt_sent` | The user's first typed message in an onboarding session. Slash commands are excluded, so the hidden `/onboard-connect` submission does not count. |
64+
| `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `big_pickle_confirm`, `gateway_auth`, or `connected`. Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. |
65+
66+
### A note on the derived activation events
67+
68+
`activation_menu_shown`, `activation_job_selected`, and `first_job_completed` are **inferred, not observed**. The activation menu is not a UI element: it is text the model writes from a prompt template, and the user picks a job by replying in free text. Nothing in the CLI can see either moment directly.
69+
70+
They are therefore inferred from the closest deterministic signals — the menu from the command dispatch or the completed environment scan, the job from the first matching tool or skill invocation that follows. Treat the counts as **lower bounds**, and note two specific gaps:
71+
72+
- The "something else" branch has no tool signature at all and is never counted.
73+
- `first_job_completed` only fires for jobs with a real completion signal. Skill-driven jobs (downstream impact, SQL review, cost) load an instruction bundle and then do their work through other tools, so their completion is not observable and they are absent from this event rather than wrongly counted in it.
74+
75+
Each event includes a timestamp, anonymous session ID, a per-launch correlation ID (`launch_id` — a random value regenerated every process start, not persisted and not derived from your machine or identity; it exists only to group events from the same run), CLI version, and an anonymous machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and never tied to any personal information).
5276

5377
## Delivery & Reliability
5478

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 54 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1351,6 +1351,22 @@ export namespace Telemetry {
13511351
// above can consult it. See shutdown() for why concurrent shutdowns must be serialized.
13521352
let shutdownPromise: Promise<void> | undefined
13531353

1354+
// altimate_change start — per-launch correlation id, shared across threads via the environment.
1355+
// The TUI worker is spawned after the CLI middleware has already initialised telemetry on the
1356+
// main thread, and a Worker inherits a copy of process.env, so whichever thread runs first
1357+
// publishes the value and the other reads it. Lazy rather than module-init so importing the
1358+
// telemetry module (in tests, tooling) does not mint ids nobody uses.
1359+
const LAUNCH_ID_ENV = "ALTIMATE_LAUNCH_ID"
1360+
1361+
function launchId(): string {
1362+
const existing = process.env[LAUNCH_ID_ENV]
1363+
if (existing) return existing
1364+
const created = randomUUID()
1365+
process.env[LAUNCH_ID_ENV] = created
1366+
return created
1367+
}
1368+
// altimate_change end
1369+
13541370
function parseConnectionString(cs: string): AppInsightsConfig | undefined {
13551371
const parts: Record<string, string> = {}
13561372
for (const segment of cs.split(";")) {
@@ -1383,6 +1399,12 @@ export namespace Telemetry {
13831399
source: clientSource,
13841400
project_id: fields.project_id ?? projectId,
13851401
...(machineId && { machine_id: machineId }),
1402+
// altimate_change — groups every event from one process launch. The onboarding funnel
1403+
// spans the TUI main thread and the server worker, and most of it runs before any chat
1404+
// session exists, so `session_id` is empty for the first half and real for the second —
1405+
// leaving no key to join a single run on. This is not persisted, not derived from the
1406+
// machine or the user, and not reused across launches; it only says "same run".
1407+
launch_id: launchId(),
13861408
}
13871409
const measurements: Record<string, number> = {}
13881410

@@ -1485,9 +1507,22 @@ export namespace Telemetry {
14851507
try {
14861508
machineId = fs.readFileSync(machineIdPath, "utf8").trim()
14871509
} catch {
1488-
machineId = randomUUID()
1510+
// altimate_change start — create exclusively so two threads cannot mint different ids.
1511+
// The TUI main thread and the server worker each initialise their own copy of this
1512+
// module, and on a genuinely new install both can find the file missing at the same
1513+
// moment. With a plain write, the loser's value overwrites the winner's while both keep
1514+
// their own in memory, so a single first run reports two machine_ids — breaking the
1515+
// fallback identity exactly on the run that matters most. `wx` makes one of them fail,
1516+
// and the loser re-reads what the winner wrote.
1517+
const candidate = randomUUID()
14891518
fs.mkdirSync(path.dirname(machineIdPath), { recursive: true })
1490-
fs.writeFileSync(machineIdPath, machineId, "utf8")
1519+
try {
1520+
fs.writeFileSync(machineIdPath, candidate, { encoding: "utf8", flag: "wx" })
1521+
machineId = candidate
1522+
} catch {
1523+
machineId = fs.readFileSync(machineIdPath, "utf8").trim()
1524+
}
1525+
// altimate_change end
14911526
}
14921527
} catch {
14931528
// Machine ID unavailable — proceed without it
@@ -1604,7 +1639,23 @@ export namespace Telemetry {
16041639
// does NOT cancel it: the losing promise keeps running and resets module state after the
16051640
// caller has already moved on. Exit paths pass a budget so the fetch itself is aborted.
16061641
export async function shutdown(opts?: { timeoutMs?: number }) {
1607-
if (shutdownPromise) return shutdownPromise
1642+
if (shutdownPromise) {
1643+
// An earlier caller is mid-flush, possibly on the default 10s budget. Returning its promise
1644+
// unchanged would silently ignore this caller's deadline and make the exit path wait past
1645+
// it — after which the worker is terminated anyway and the buffer is lost regardless. We
1646+
// cannot shorten the in-flight flush, but we can stop making the caller wait for it.
1647+
const budget = opts?.timeoutMs
1648+
if (budget === undefined) return shutdownPromise
1649+
let timer: ReturnType<typeof setTimeout> | undefined
1650+
await Promise.race([
1651+
shutdownPromise,
1652+
new Promise<void>((resolve) => {
1653+
timer = setTimeout(resolve, budget)
1654+
}),
1655+
]).catch(() => {})
1656+
if (timer) clearTimeout(timer)
1657+
return
1658+
}
16081659
shutdownPromise = doShutdown(opts?.timeoutMs).finally(() => {
16091660
shutdownPromise = undefined
16101661
})

‎packages/opencode/src/altimate/telemetry/onboarding.ts‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,10 @@ import { Telemetry } from "./index"
2929
export const ONBOARDING_STAGES = [
3030
"started",
3131
"model_picker",
32+
"provider_setup",
3233
"big_pickle_confirm",
3334
"gateway_auth",
3435
"connected",
35-
"scan_gate",
36-
"activation",
3736
] as const
3837

3938
export type OnboardingStage = (typeof ONBOARDING_STAGES)[number]
@@ -83,12 +82,17 @@ let funnelStarted = false
8382
const STAGE_FOR_EVENT: Partial<Record<OnboardingEventInput["type"], OnboardingStage>> = {
8483
onboarding_started: "started",
8584
model_picker_shown: "model_picker",
85+
// Any provider choice enters setup. Without this, a user who picks Anthropic and quits during
86+
// key entry is reported as abandoning at `model_picker` — as if they never chose anything.
87+
provider_selected: "provider_setup",
8688
big_pickle_confirm_shown: "big_pickle_confirm",
8789
gateway_device_code_issued: "gateway_auth",
8890
instance_connected: "connected",
8991
onboarding_completed: "connected",
90-
scan_gate_shown: "scan_gate",
91-
activation_menu_shown: "activation",
92+
// Deliberately no entry for scan_gate_shown / activation_menu_shown. Abandonment is defined as
93+
// quitting BEFORE connecting, and both of those only happen after `onboarding_completed` has
94+
// set `completed`, which suppresses abandonment entirely. Stages for them would be values that
95+
// can never be reported.
9296
}
9397

9498
function advance(stage: OnboardingStage) {

0 commit comments

Comments
 (0)