Skip to content

Commit cf46d65

Browse files
anandgupta42claude
andcommitted
fix: honor the documented telemetry.disabled config opt-out; harden first-run gates
Release-review fixes for v0.11.1 (multi-persona review + Codex review), all shipped in this release instead of as follow-ups. - `ConfigV1.Info` now declares `telemetry: { disabled?: boolean }`. The public telemetry docs have promised this opt-out, but the schema never had the field, so the strict top-level key check rejected any config file that set it: that file failed to parse (taking its other settings with it) and `Telemetry.doInit()` treated the failure as "proceed enabled". - `cli/tui/worker.ts`: `Telemetry.init()` moved from module top level (no Instance on the ALS, so `Config.get()` threw and the opt-out was bypassed for the first session) into `Instance.restore(ctx, ...)` in `traceReady`. The stall monitor still starts at boot; its events buffer until init drains them. - `ConfigPlugin.needsDependencies`: guard non-string plugin specs before `startsWith` so a malformed tuple cannot throw inside the config layer. - `Telemetry.loopStallFor`: negated threshold check so a NaN lag yields no event. - docs: add `startup_ready`, `event_loop_stall`, `altimate_base_registration` rows to `docs/docs/reference/telemetry.md` (per its own "Adding a New Event" checklist); refresh the stale event count in `docs/docs/llms.txt`. - tests: `test/telemetry/config-opt-out.test.ts` proves the opt-out end to end inside an Instance context. Deferred findings filed as #1296, #1297, #1298, #1299, #1300. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_012uzYijGHdhxvpUzhgWxayo
1 parent f4036a8 commit cf46d65

7 files changed

Lines changed: 180 additions & 10 deletions

File tree

‎docs/docs/llms.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
- [Agent Skills](https://help.altimate.ai/code/configure/skills/): How to configure, discover, and add custom skills.
3535
- [Permissions](https://help.altimate.ai/code/configure/permissions/): Permission levels, pattern matching, per-agent restrictions, deny rules for destructive SQL.
3636
- [Tracing](https://help.altimate.ai/code/configure/trace/): Local-first observability covering trace schema, span types, live viewing, remote OTLP exporters, and crash recovery.
37-
- [Telemetry](https://help.altimate.ai/code/reference/telemetry/): 25 anonymized event types, privacy guarantees, opt-out instructions.
37+
- [Telemetry](https://help.altimate.ai/code/reference/telemetry/): 60+ anonymized event types, privacy guarantees, opt-out instructions.
3838

3939
## Reference
4040

‎docs/docs/reference/telemetry.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@ We collect the following categories of events:
4343
| `sql_execute_failure` | A SQL execution fails (warehouse type, query type, error message, PII-masked SQL — no raw values) |
4444
| `core_failure` | An internal tool error occurs (tool name, category, error class, truncated error message, PII-safe input signature, and optionally masked arguments — no raw values or credentials) |
4545
| `first_launch` | Fired once on the first CLI run after an install or upgrade, triggered by a marker file the installer wrote — the installers themselves send nothing and contact no telemetry endpoint. Contains the installed version, `is_upgrade`, and `install_method` (`curl`, `powershell`, `npm`, `vscode-extension`, `local` for `install --binary`, or `unknown` for markers written before the field existed). `vscode-extension` starts appearing only once an extension build containing the marker write ships, so a zero share for it means the extension has not rolled out yet rather than no extension installs. No PII. **Reading `is_upgrade`:** it means "this machine had run altimate-code before", probed as whether `~/.altimate/machine-id` already existed — *not* "a binary was already present". A reinstall onto a machine that ever ran the CLI reports `is_upgrade: true`, and `altimate uninstall` leaves `machine-id` in place, so a metric excluding upgrades counts installs **per previously-unseen machine** and undercounts reinstalls onto known ones. (`is_upgrade` is a boolean in the event schema; it arrives in Application Insights `customDimensions` as a string, so KQL filters read `tostring(customDimensions.is_upgrade) != "true"`.) Delivery is at-most-once: the marker is deleted before the event flushes, so a process that dies first loses that install rather than re-firing it every launch. Local `--binary` installs report `version: "local"`. |
46+
| `startup_ready` | Fired once per process when the top-level command can serve its first request or frame (`serve` listening, TUI transport resolved, `run` about to send its first prompt). Contains the command name, `duration_ms` since process start, and `fresh_install` (true when this process also emitted a non-upgrade `first_launch`). No PII. |
47+
| `event_loop_stall` | A 250 ms monitor tick fired more than 1 s late, meaning the event loop was blocked for that long (for example an in-process dependency install). Contains the command name, `thread` (`main` or `worker`), `blocked_ms`, and `since_start_ms`. Capped at 20 per thread (the main thread and the TUI server worker each keep their own counter). No PII. |
48+
| `altimate_base_registration` | Timing and outcome of each Altimate Base registration after consent (concurrent calls share one result, and a still-valid cached credential reports `success` without a network round trip): `result` (`success`, `network`, `http`, `response`, `cancelled`, `configuration`, or `error`), `duration_ms`, and the HTTP `status` when the result is `http`. Distinct from `altimate_base_register_result`, which records the onboarding-flow outcome. No gateway URL, response body, credential, or error text is included. |
4649
| `task_outcome_signal` | Behavioral quality signal at session end — accepted, error, abandoned, or cancelled. Includes tool count, step count, duration, and last tool category. No user content. |
4750
| `task_classified` | Intent classification of the first user message using keyword matching — category (e.g. `debug_dbt`, `write_sql`, `optimize_query`), confidence score, and detected warehouse type. No user text is sent — only the classified category. |
4851
| `tool_chain_outcome` | Aggregated tool execution sequence at session end — ordered tool names (capped at 50), error count, recovery count, final outcome, duration, and cost. No tool arguments or outputs. |

‎packages/core/src/v1/config/config.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,18 @@ export const Info = Schema.Struct({
8888
username: Schema.optional(Schema.String).annotate({
8989
description: "Custom username to display in conversations instead of system username",
9090
}),
91+
// altimate_change start — the documented telemetry opt-out (docs/docs/reference/telemetry.md).
92+
// Telemetry.doInit() reads `telemetry.disabled` from the merged config; without this field the
93+
// strict top-level key check rejected any config file that set it, so the file failed to parse
94+
// and telemetry fell open. Declared here so the opt-out is a real, schema-visible setting.
95+
telemetry: Schema.optional(
96+
Schema.Struct({
97+
disabled: Schema.optional(Schema.Boolean).annotate({
98+
description: "Disable all anonymous usage telemetry (equivalent to ALTIMATE_TELEMETRY_DISABLED=true)",
99+
}),
100+
}),
101+
).annotate({ description: "Anonymous usage telemetry settings" }),
102+
// altimate_change end
91103
mode: Schema.optional(
92104
Schema.StructWithRest(
93105
Schema.Struct({ build: Schema.optional(ConfigAgentV1.Info), plan: Schema.optional(ConfigAgentV1.Info) }),

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2151,7 +2151,9 @@ export namespace Telemetry {
21512151
thread: "main" | "worker",
21522152
): Event | undefined {
21532153
const lag = now - expectedAt
2154-
if (lag <= thresholdMs) return undefined
2154+
// Negated form so a NaN lag (never expected from performance.now, but cheap to exclude) is
2155+
// treated as "no stall" instead of emitting an event with NaN fields.
2156+
if (!(lag > thresholdMs)) return undefined
21552157
return {
21562158
type: "event_loop_stall",
21572159
timestamp: Date.now(),

‎packages/opencode/src/cli/tui/worker.ts‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,11 @@ import { Telemetry } from "@/altimate/telemetry"
2929
import * as OnboardingTelemetry from "@/altimate/telemetry/onboarding"
3030
// altimate_change start — first-run health: this thread does not initialise telemetry until the
3131
// first prompt, but config and plugin loading (the in-process arborist install that froze fresh
32-
// installs) run here before that. Start the stall monitor at boot and initialise telemetry on this
33-
// thread right away (idempotent, same as the CLI middleware does on the main thread): without the
34-
// init, stall events sit in a pre-init buffer that shutdown discards, so a user who waits through
35-
// the freeze and quits before the first prompt would never record it. init() drains buffered
36-
// anchor events as soon as it enables.
32+
// installs) run here before that. Start the stall monitor at boot so those stalls are captured;
33+
// its events buffer until Telemetry.init() runs inside the Instance context in traceReady below
34+
// (NOT here: at module top level Config.get() throws and doInit() proceeds as enabled, which
35+
// silently bypassed a `telemetry.disabled` config opt-out for the whole first session).
3736
Telemetry.startLoopMonitor()
38-
Telemetry.init().catch(() => {})
3937
// altimate_change end
4038
// altimate_change start — heal the datamate MCP entry at boot. `altimate serve` runs
4139
// this sync before listening (cli/cmd/serve.ts), but the TUI worker never did, so an
@@ -81,7 +79,15 @@ const traceReady: Promise<void> = (async () => {
8179
// altimate_change end
8280
try {
8381
const ctx = await InstanceRuntime.load({ directory: process.cwd() })
84-
await Instance.restore(ctx, () => traceConsumer.loadConfig())
82+
await Instance.restore(ctx, () => {
83+
// altimate_change start — first-run health: initialise telemetry here, inside the Instance
84+
// context, so doInit()'s Config.get() can see a `telemetry.disabled` opt-out. init() is
85+
// idempotent and drains the anchor events (event_loop_stall) buffered since boot as soon as
86+
// it enables; if the instance fails to load, the first prompt initialises telemetry as before.
87+
Telemetry.init().catch(() => {})
88+
// altimate_change end
89+
return traceConsumer.loadConfig()
90+
})
8591
} catch {
8692
await traceConsumer.loadConfig().catch(() => {})
8793
}

‎packages/opencode/src/config/plugin.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,8 @@ export function needsDependencies(dir: string, plugins: readonly ConfigPluginV1.
6161
}
6262
return (plugins ?? []).some((plugin) => {
6363
const spec = pluginSpecifier(plugin)
64-
if (!spec.startsWith("file://")) return false
64+
// A parsed config only yields string specs; guard anyway so a malformed tuple cannot throw here.
65+
if (typeof spec !== "string" || !spec.startsWith("file://")) return false
6566
try {
6667
const file = fileURLToPath(spec)
6768
try {
Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
// altimate_change — regression test for two fixes that together make the `telemetry.disabled`
2+
// config opt-out actually work.
3+
//
4+
// Fix 1 (src/cli/tui/worker.ts): `Telemetry.init()` used to run at module top level in the bare
5+
// TUI worker, where `Config.get()` throws (no Instance context yet), so `doInit()` proceeded as
6+
// enabled and the opt-out was silently bypassed for the whole first session. It now runs inside
7+
// `Instance.restore(ctx, ...)` in `traceReady` (worker.ts's `traceReady` IIFE), specifically so
8+
// `doInit()`'s `Config.get()` can see the merged, instance-scoped config.
9+
//
10+
// Fix 2 (packages/core/src/v1/config/config.ts): even with fix 1, `ConfigV1.Info` (the schema
11+
// every project/global/local config file is parsed against) had NO `telemetry` field at all.
12+
// `doInit()` (packages/opencode/src/altimate/telemetry/index.ts:2019-2020) reads
13+
// `(await Config.get() as any).telemetry?.disabled` — before this fix the `as any` cast was hiding
14+
// that it read a field the schema never declared. Writing `{ "telemetry": { "disabled": true } }`
15+
// to a config file made THAT FILE's `ConfigParse.schema(ConfigV1.Info, ...)` call throw
16+
// `unrecognized_keys`, which propagated out of `Config.get()` as a `ConfigInvalidError` Die defect
17+
// instead of merging past that file — `doInit()`'s own try/catch around `Config.get()`
18+
// (index.ts:2017-2023) treats ANY failure as "Config unavailable — proceed with telemetry
19+
// enabled", so the opt-out did not merely fail silently: adding it to a config file broke that
20+
// file's config loading entirely (for every other config-dependent feature in that Instance too,
21+
// not just telemetry) and telemetry stayed on regardless. `ConfigV1.Info` now declares
22+
// `telemetry: Schema.optional(Schema.Struct({ disabled: Schema.optional(Schema.Boolean) }))`, so
23+
// the field parses cleanly and `doInit()` can actually see it.
24+
//
25+
// Both fixes are necessary: fix 1 alone gets `Config.get()` a working Instance context but the
26+
// schema still rejected the field; fix 2 alone adds the field to the schema but the bare worker's
27+
// `Config.get()` call still would have thrown before ever reading it. The three tests below prove,
28+
// in order: the Instance-context mechanism resolves the right config (fix 1), the happy path with
29+
// no opt-out still enables telemetry, and the opt-out itself now disables it end to end (fix 2).
30+
import { afterEach, describe, expect, spyOn } from "bun:test"
31+
import { Effect, Layer } from "effect"
32+
import { FSUtil } from "@opencode-ai/core/fs-util"
33+
import { Config } from "@/config/config"
34+
import { Telemetry } from "@/altimate/telemetry"
35+
import { TestInstance } from "../fixture/fixture"
36+
import { testEffect } from "../lib/effect"
37+
38+
const it = testEffect(Layer.mergeAll(Config.defaultLayer, FSUtil.defaultLayer))
39+
40+
// Local copy of config.test.ts's / tui.test.ts's env-restore helper — neither file exports it, and
41+
// every existing suite that needs one duplicates it rather than centralizing.
42+
function withProcessEnvs<A, E, R>(entries: Record<string, string | undefined>, effect: Effect.Effect<A, E, R>) {
43+
return Effect.acquireUseRelease(
44+
Effect.sync(() => {
45+
const originals: Record<string, string | undefined> = {}
46+
for (const [key, value] of Object.entries(entries)) {
47+
originals[key] = process.env[key]
48+
if (value === undefined) delete process.env[key]
49+
else process.env[key] = value
50+
}
51+
return originals
52+
}),
53+
() => effect,
54+
(originals) =>
55+
Effect.sync(() => {
56+
for (const [key, original] of Object.entries(originals)) {
57+
if (original !== undefined) process.env[key] = original
58+
else delete process.env[key]
59+
}
60+
}),
61+
)
62+
}
63+
64+
// Explicit connection string bypasses isAutomatedRun()'s implicit-production-sink withholding (see
65+
// doInit()'s comment: "Telemetry's own tests set APPLICATIONINSIGHTS_CONNECTION_STRING explicitly
66+
// and are unaffected"), and both disable env vars must be cleared so only the config path is live.
67+
const TELEMETRY_ENV = {
68+
APPLICATIONINSIGHTS_CONNECTION_STRING: "InstrumentationKey=config-opt-out-test;IngestionEndpoint=https://example.com",
69+
ALTIMATE_TELEMETRY_DISABLED: undefined,
70+
OPENCODE_DISABLE_TELEMETRY: undefined,
71+
}
72+
73+
describe("Telemetry.init() inside an Instance context — config.telemetry.disabled opt-out", () => {
74+
afterEach(async () => {
75+
await Telemetry.shutdown()
76+
})
77+
78+
it.instance(
79+
"Config.get(), called the same way doInit() calls it, resolves the Instance-scoped project config instead of throwing — the mechanism the worker.ts fix relies on",
80+
() =>
81+
Effect.gen(function* () {
82+
const fs = yield* FSUtil.Service
83+
const test = yield* TestInstance
84+
yield* fs.writeJson(`${test.directory}/opencode.json`, {
85+
$schema: "https://opencode.ai/config.json",
86+
username: "instance-scoped-marker",
87+
})
88+
// The plain async Config.get() facade is exactly what doInit() calls; invoking it via
89+
// Effect.promise from inside this it.instance's Instance/InstanceRef context mirrors
90+
// Instance.restore(ctx, () => Telemetry.init()) in worker.ts's traceReady. Before the fix,
91+
// calling this at bare module scope (no Instance) would throw instead of resolving.
92+
const config = yield* Effect.promise(() => Config.get())
93+
expect((config as any).username).toBe("instance-scoped-marker")
94+
}),
95+
)
96+
97+
it.instance(
98+
"without a telemetry opt-out, Telemetry.init() run inside the Instance context enables telemetry and flushes anchor events",
99+
() =>
100+
withProcessEnvs(
101+
TELEMETRY_ENV,
102+
Effect.gen(function* () {
103+
const flushSpy = spyOn(Telemetry, "flush")
104+
try {
105+
yield* Effect.promise(() => Telemetry.init())
106+
expect(Telemetry.isEnabled()).toBe(true)
107+
Telemetry.track({
108+
type: "startup_ready",
109+
timestamp: Date.now(),
110+
session_id: "config-opt-out-enabled",
111+
command: "test",
112+
duration_ms: 1,
113+
fresh_install: false,
114+
})
115+
expect(flushSpy).toHaveBeenCalled()
116+
} finally {
117+
flushSpy.mockRestore()
118+
}
119+
}),
120+
),
121+
)
122+
123+
it.instance(
124+
"telemetry.disabled: true in project config disables telemetry end to end, with no flush",
125+
() =>
126+
withProcessEnvs(
127+
TELEMETRY_ENV,
128+
Effect.gen(function* () {
129+
const fs = yield* FSUtil.Service
130+
const test = yield* TestInstance
131+
const flushSpy = spyOn(Telemetry, "flush")
132+
try {
133+
yield* fs.writeJson(`${test.directory}/opencode.json`, {
134+
$schema: "https://opencode.ai/config.json",
135+
telemetry: { disabled: true },
136+
})
137+
yield* Effect.promise(() => Telemetry.init())
138+
expect(Telemetry.isEnabled()).toBe(false)
139+
expect(flushSpy).not.toHaveBeenCalled()
140+
} finally {
141+
flushSpy.mockRestore()
142+
}
143+
}),
144+
),
145+
)
146+
})

0 commit comments

Comments
 (0)