Skip to content

Commit fe6ac08

Browse files
sahrizviclaude
andcommitted
fix(onboarding): address bot review findings on #1049
`cubic` (18) and `kilo-code-bot` (1). Four of these are regressions from the consensus-review fixes in `8e369a8`. Correctness: - `app.tsx`: completing setup from the prompt gate before sync hydration settled made the first-run effect see `onboardingReady()` and return, skipping the funnel and the scan gate entirely. `setupComplete()` discriminates that case from a genuine returning user. - `telemetry/index.ts`: a caller arriving after a shutdown settled but before the chained `doInit()` finished started a second one, replacing the flush timer. The chained promise is now published as `initPromise` too. - `telemetry/index.ts`: `doShutdown()` cleared `cachedLaunchId`, so a long-lived `serve` process minted a new `launch_id` per prompt. Replaced with a `resetLaunchIdForTest()` seam. - `plugin/onboarding-telemetry.ts`: `activation_job_selected` is claimed in `tool.execute.before`, so a job that throws still records the selection the docs promise. - `session/prompt.ts`: an attachment-only message counted as `first_prompt_sent`. - `app.tsx` / `altimate-onboarding.tsx`: `Escape` and click-away closes recorded no `scan_gate_choice` / `big_pickle_choice` — only the inline `esc` control did. - `dialog-model.tsx`: `via_search` was dropped when returning from Big Pickle. - `context/onboarding-telemetry.tsx`: `try`/`catch` cannot catch an async tracker's rejection. Bounds and privacy: - `cmd/tui.ts`: the exit budget covered only the flush, not the credential read or `shutdown()`'s await of initialization. - `telemetry/index.ts`: `shutdown()` awaited an in-flight flush unbounded (10s). - `worker.ts` / `cmd/tui.ts`: the two halves of the exit budget now share `Telemetry.TUI_SHUTDOWN_BUDGET_MS`. - `sample-setup.ts`: the `sample_source_missing` output embedded absolute paths. Tests: activation-event tests drive both tool hooks rather than `.after` alone, plus coverage for a tool that throws. The launch-correlation test redirects `HOME` so real `init()` no longer writes `~/.altimate/machine-id`. Declined: the stale `commandSubmission` flag after a failed slash command. There is no `command.execute.after` hook to clear it from, it suppresses at most one event in a session where the command already failed, and dropping the guard would mis-record the scan gate's synthetic `/onboard-connect` as a typed prompt in every fresh onboarding. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
1 parent 1c21544 commit fe6ac08

15 files changed

Lines changed: 259 additions & 88 deletions

File tree

‎docs/docs/reference/telemetry.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ We collect the following categories of events:
6161
| `activation_menu_shown` | The activation menu was (very likely) rendered. `variant` is `warehouse` or `no_data`. **Derived** — see the note below. |
6262
| `activation_job_selected` / `first_job_completed` | Which activation job the user started and, where observable, finished. Completion is reported only for the job that was actually selected, so the two form a coherent pair. **Derived** — see the note below. |
6363
| `first_prompt_sent` | The user's first typed message in an onboarding session. Slash commands are excluded, so the hidden `/onboard-connect` submission does not count. |
64-
| `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `big_pickle_confirm`, `gateway_auth`, or `connected`. Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. |
64+
| `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `big_pickle_confirm`, or `gateway_auth`. (`connected` is a funnel position but never a `last_stage` — reaching it means the run completed, which is not an abandonment.) Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. |
6565

6666
### A note on the derived activation events
6767

‎packages/opencode/src/altimate/plugin/onboarding-telemetry.ts‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,20 @@ export async function OnboardingTelemetryPlugin(_input: PluginInput): Promise<Ho
103103
}
104104
},
105105

106+
"tool.execute.before": async (input, output) => {
107+
if (!OnboardingTelemetry.isOnboardingSession(input.sessionID)) return
108+
// Selection lands on the first job-shaped tool call, and it is claimed HERE rather than in
109+
// tool.execute.after because the docs promise that a job which started and then failed
110+
// still counts as selected — the user did choose it. Claiming it after the tool returned
111+
// dropped exactly those cases (skill lookup, permission denial, execution error), which are
112+
// the ones a funnel most needs to see. Completion detection stays in .after.
113+
const job = jobForTool(input.tool, output.args)
114+
if (!job) return
115+
if (OnboardingTelemetry.claimActivationJobSelected(input.sessionID, job)) {
116+
void OnboardingTelemetry.emit({ type: "activation_job_selected", job }, input.sessionID)
117+
}
118+
},
119+
106120
"tool.execute.after": async (input, output) => {
107121
if (!OnboardingTelemetry.isOnboardingSession(input.sessionID)) return
108122

@@ -126,11 +140,6 @@ export async function OnboardingTelemetryPlugin(_input: PluginInput): Promise<Ho
126140
const job = jobForTool(input.tool, input.args)
127141
if (!job) return
128142

129-
// Selection lands on the first job-shaped tool call. A job that starts and then fails still
130-
// counts as selected — the user did choose it.
131-
if (OnboardingTelemetry.claimActivationJobSelected(input.sessionID, job)) {
132-
void OnboardingTelemetry.emit({ type: "activation_job_selected", job }, input.sessionID)
133-
}
134143
// Only complete the job that was actually selected. Skill-driven jobs stay without a
135144
// completion event, as the docs already state.
136145
if (

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 50 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,21 @@ export namespace Telemetry {
5050
const MAX_BUFFER_SIZE = 200
5151
const REQUEST_TIMEOUT_MS = 10_000
5252

53+
/**
54+
* altimate_change — single source of truth for the TUI exit-path flush budget, referenced by
55+
* BOTH the `withTimeout(client.call("shutdown", ...))` deadline on the main thread
56+
* (cli/cmd/tui.ts) and the worker's remaining-budget computation (cli/tui/worker.ts). They must
57+
* agree: the worker's buffer flush exists only because `worker.terminate()` would otherwise
58+
* discard it, and if the main thread's deadline were the shorter of the two it would truncate
59+
* the flush and silently restore the data loss this guards against.
60+
*/
61+
export const TUI_SHUTDOWN_BUDGET_MS = 5000
62+
63+
/** Exit-path budget for the MAIN thread's own flush. Strictly smaller than the worker's,
64+
* because it runs after the worker shutdown RPC has already returned and still has to finish
65+
* before `process.exit(0)`. */
66+
export const EXIT_FLUSH_BUDGET_MS = 2000
67+
5368
export type Event =
5469
// altimate_change start — add os/arch/node_version for environment segmentation
5570
| {
@@ -1446,6 +1461,11 @@ export namespace Telemetry {
14461461

14471462
let cachedLaunchId: string | undefined
14481463

1464+
/** Test seam — the cache is intentionally process-lifetime, so shutdown() does not clear it. */
1465+
export function resetLaunchIdForTest() {
1466+
cachedLaunchId = undefined
1467+
}
1468+
14491469
export function launchId(): string {
14501470
// The worker reads the value the TUI handed it through WorkerOptions.env; the main thread
14511471
// generates it. Cached in module scope rather than written back to process.env — the worker
@@ -1557,12 +1577,19 @@ export namespace Telemetry {
15571577
// Chaining onto shutdownPromise keeps the generations separate: callers arriving mid-shutdown
15581578
// wait for the new init rather than joining the dying one.
15591579
if (shutdownPromise) {
1560-
return (reinitPromise ??= shutdownPromise
1561-
.catch(() => {})
1562-
.then(doInit)
1563-
.finally(() => {
1564-
reinitPromise = undefined
1565-
}))
1580+
if (!reinitPromise) {
1581+
reinitPromise = shutdownPromise
1582+
.catch(() => {})
1583+
.then(doInit)
1584+
.finally(() => {
1585+
reinitPromise = undefined
1586+
})
1587+
// Publish it as initPromise too. Without this, a caller arriving after the shutdown had
1588+
// settled — but before this chained doInit() finished — saw an empty initPromise and
1589+
// started a SECOND one, replacing the flush timer and racing over the buffer.
1590+
initPromise = reinitPromise
1591+
}
1592+
return reinitPromise
15661593
}
15671594
return (initPromise ??= doInit())
15681595
// altimate_change end
@@ -1803,7 +1830,19 @@ export namespace Telemetry {
18031830
}
18041831
// altimate_change — drain any flush already running before the final one, so a timer batch
18051832
// in mid-request is not abandoned and cannot write back into a buffer we are about to clear.
1806-
if (inFlightFlush) await inFlightFlush.catch(() => {})
1833+
if (inFlightFlush) {
1834+
// Bounded: a flush already in flight uses the DEFAULT request timeout, so awaiting it
1835+
// unbounded could burn 10s before the bounded flush below even starts — well past the 5s
1836+
// the TUI allows the whole shutdown RPC.
1837+
let drainTimer: ReturnType<typeof setTimeout> | undefined
1838+
await Promise.race([
1839+
inFlightFlush.catch(() => {}),
1840+
new Promise<void>((resolve) => {
1841+
drainTimer = setTimeout(resolve, timeoutMs ?? REQUEST_TIMEOUT_MS)
1842+
}),
1843+
])
1844+
if (drainTimer) clearTimeout(drainTimer)
1845+
}
18071846
await flush(timeoutMs)
18081847
inFlightFlush = undefined
18091848
enabled = false
@@ -1813,9 +1852,10 @@ export namespace Telemetry {
18131852
sessionId = ""
18141853
projectId = ""
18151854
machineId = ""
1816-
// altimate_change — the launch id is per process, but tests re-init the module in one process;
1817-
// leaving it cached made a second lifecycle inherit the first one's id.
1818-
cachedLaunchId = undefined
1855+
// NOTE: cachedLaunchId is deliberately NOT cleared here. session/prompt.ts shuts telemetry
1856+
// down at the end of every session, so clearing it would mint a fresh launch_id per prompt in
1857+
// a long-lived `serve` process and shatter the per-launch correlation it exists to provide.
1858+
// Tests use resetLaunchIdForTest() instead.
18191859
// altimate_change — only clear initPromise if it is still the one this shutdown began with.
18201860
// init() can set `initPromise = shutdownPromise.then(doInit)`; nulling that unconditionally
18211861
// discards a doInit() which has not run yet, so the next init() starts a second one.

‎packages/opencode/src/altimate/telemetry/onboarding.ts‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@ export const ONBOARDING_STAGES = [
3232
"provider_setup",
3333
"big_pickle_confirm",
3434
"gateway_auth",
35+
// NOTE: reaching this stage means the run completed, and emitAbandonedIfIncomplete() returns
36+
// early on `completed`. So "connected" is a valid funnel position but never a `last_stage` on
37+
// an abandonment — see the enum note in docs/docs/reference/telemetry.md.
3538
"connected",
3639
] as const
3740

@@ -150,11 +153,6 @@ export function markStage(stage: OnboardingStage) {
150153
advance(stage)
151154
}
152155

153-
/** True once `onboarding_completed` has been emitted on this thread. */
154-
export function isCompleted() {
155-
return completed
156-
}
157-
158156
/**
159157
* Emit `onboarding_abandoned` if the user got somewhere in the funnel and never completed.
160158
* Call on the exit path, before the final flush. No-ops when the user never started (a

‎packages/opencode/src/altimate/tools/project-scan.ts‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -936,9 +936,6 @@ export const ProjectScanTool = Tool.define("project_scan", {
936936
// for. Deliberately separate from `environment_census` above, which stays the richer
937937
// dbt/warehouse fingerprint; this one answers "did the scan find the user a working setup".
938938
//
939-
// Fires on every project_scan, not just the onboarding one — the tool is also reachable via
940-
// /discover and any model-initiated call.
941-
//
942939
// has_warehouse and connections_found both come from totalConnections, NOT
943940
// connections.alreadyConfigured: connections discovered from dbt profiles, docker, and env
944941
// vars are counted separately, and a user whose only warehouse was auto-discovered would

‎packages/opencode/src/altimate/tools/sample-setup.ts‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -109,13 +109,16 @@ export const SampleSetupTool = Tool.define("sample_setup", {
109109
`Could not locate the shipped starter sample source. This usually means the CLI ` +
110110
`was installed without its wrapper package assets. Reinstall with: ` +
111111
`\`npm i -g @altimateai/altimate-code@latest\`\n\n` +
112-
`Underlying error: ${message}`
112+
// Redacted for the same reason as the materialize_failed branch below: sample-manifest
113+
// read/resolve errors embed absolute install paths, and `output` is what the model sees
114+
// and what gets replayed to the provider on later turns.
115+
`Underlying error: ${redactPaths(message)}`
113116
// altimate_change — onboarding funnel: a broken install (sample assets missing) is a
114117
// sample-setup failure too. Without this the failure count only covers materialization
115118
// errors and silently under-reports the "CLI shipped without its assets" case.
116119
if (OnboardingTelemetry.isOnboardingSession(ctx.sessionID))
117120
void OnboardingTelemetry.emit({
118-
type: "sample_setup_completed",
121+
type: "sample_setup_completed",
119122
success: false,
120123
models: 0,
121124
tables: 0,
@@ -176,7 +179,7 @@ export const SampleSetupTool = Tool.define("sample_setup", {
176179
const sampleContents = countSampleContents(sampleSource.path)
177180
if (OnboardingTelemetry.isOnboardingSession(ctx.sessionID))
178181
void OnboardingTelemetry.emit({
179-
type: "sample_setup_completed",
182+
type: "sample_setup_completed",
180183
success: true,
181184
models: sampleContents.models,
182185
tables: sampleContents.tables,
@@ -207,7 +210,7 @@ export const SampleSetupTool = Tool.define("sample_setup", {
207210
// paths (unsafe HOME, unwritable parent), so it is not sent — only the boolean.
208211
if (OnboardingTelemetry.isOnboardingSession(ctx.sessionID))
209212
void OnboardingTelemetry.emit({
210-
type: "sample_setup_completed",
213+
type: "sample_setup_completed",
211214
success: false,
212215
models: 0,
213216
tables: 0,

‎packages/opencode/src/cli/cmd/tui.ts‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ export const TuiThreadCommand = cmd({
153153
if (stopped) return
154154
stopped = true
155155
process.off("SIGUSR2", reload)
156-
await withTimeout(client.call("shutdown", undefined), 5000).catch(() => {})
156+
await withTimeout(client.call("shutdown", undefined), Telemetry.TUI_SHUTDOWN_BUDGET_MS).catch(() => {})
157157
worker.terminate()
158158
}
159159

@@ -285,13 +285,23 @@ export const TuiThreadCommand = cmd({
285285
// flush() would otherwise block for REQUEST_TIMEOUT_MS (10s) on a blackholed network — a
286286
// visible hang between the user quitting and the shell prompt returning.
287287
try {
288-
// Ask whether gateway credentials landed. The success events are emitted on the worker
289-
// thread and this state is main-thread-owned, so without this a browser sign-in that
290-
// completed just before the user quit is reported as an abandonment in the same launch
291-
// that already reported instance_connected.
292-
const connected = await AltimateApi.isConfigured().catch(() => false)
293-
await OnboardingTelemetry.emitAbandonedIfIncomplete({ connected })
294-
await Telemetry.shutdown({ timeoutMs: 2000 })
288+
// The inner bound covers the flush only. Two things ahead of it can also stall — the
289+
// credential read, and shutdown()'s await of an initialization the request middleware
290+
// started — so the whole finalizer is raced too. Racing is safe HERE (unlike inside
291+
// flush, per the note above) because the only thing after it is process.exit(0): nothing
292+
// resumes to observe module state a straggler might still be mutating.
293+
await withTimeout(
294+
(async () => {
295+
// Ask whether gateway credentials landed. The success events are emitted on the
296+
// worker thread and this state is main-thread-owned, so without this a browser
297+
// sign-in that completed just before the user quit is reported as an abandonment in
298+
// the same launch that already reported instance_connected.
299+
const connected = await AltimateApi.isConfigured().catch(() => false)
300+
await OnboardingTelemetry.emitAbandonedIfIncomplete({ connected })
301+
await Telemetry.shutdown({ timeoutMs: Telemetry.EXIT_FLUSH_BUDGET_MS })
302+
})(),
303+
Telemetry.EXIT_FLUSH_BUDGET_MS + 1000,
304+
)
295305
} catch {
296306
// Never let telemetry delay or break exit.
297307
}

‎packages/opencode/src/cli/tui/worker.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,9 @@ import { Instance } from "@/project/instance"
2727
// altimate_change — onboarding telemetry: flush this thread's buffer in rpc.shutdown()
2828
import { Telemetry } from "@/altimate/telemetry"
2929

30-
// altimate_change — must match the withTimeout budget in cli/cmd/tui.ts stop().
31-
const SHUTDOWN_BUDGET_MS = 5000
30+
// altimate_change — shared with the withTimeout budget in cli/cmd/tui.ts stop(), so the coupling
31+
// is enforced by the compiler rather than by a comment.
32+
const SHUTDOWN_BUDGET_MS = Telemetry.TUI_SHUTDOWN_BUDGET_MS
3233

3334
Heap.start()
3435

‎packages/opencode/src/session/prompt.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,9 +1124,16 @@ export namespace SessionPrompt {
11241124
// - `consumeCommandSubmission` excludes slash commands, because the scan gate submits
11251125
// a hidden `/onboard-connect` as an ordinary user message: it would otherwise be
11261126
// recorded as the user's first typed prompt in every fresh onboarding.
1127+
// - a real text part, because a message carrying only attachments (a dragged-in file,
1128+
// a pasted image) reached this block too and was recorded as the user's first typed
1129+
// prompt before they had typed anything. Same filter as the intent classifier above.
11271130
const fromCommand = OnboardingTelemetry.consumeCommandSubmission(sessionID)
1131+
const hasUserText = !!userMsg?.parts.some(
1132+
(p) => p.type === "text" && !p.ignored && !p.synthetic && p.text.trim().length > 0,
1133+
)
11281134
if (
11291135
!fromCommand &&
1136+
hasUserText &&
11301137
OnboardingTelemetry.isOnboardingSession(sessionID) &&
11311138
OnboardingTelemetry.claimFirstPrompt(sessionID)
11321139
) {

0 commit comments

Comments
 (0)