Skip to content

Commit 4f464dc

Browse files
vanceingallsclaude
andcommitted
feat(cli,studio): telemetry opt-out is canary opt-out
Both reviewers flagged the same gap: seed injection was gated on telemetryShouldTrack(), but canary EVALUATION was not. An install with DO_NOT_TRACK=1 was still bucketed and still had real code paths flipped (e.g. HF_DE_PARALLEL_ROUTER), silently and unmeasurably. A canary is a measured rollout — we enrol a slice precisely so it can be compared against everyone else. An install that sends nothing can't be compared, so enrolling it buys no signal and only changes that user's code path, on an experimental feature, without their knowledge. That is the wrong side of an opt-out. Resolves to a new `telemetry_opt_out` reason BEFORE bucketing, so no cohort is assigned at all. Distinct from `excluded` because "why is my canary off" has a very different answer for CI than for opted-out, and the reason never reaches telemetry by construction. Covers every opt-out route: persisted preference, the runtime env vars and dev/telemetry-disabled builds via policy.ts, and Studio's hyperframes-studio:telemetryDisabled. An explicit HF_CANARY_* / ?hf_canary_*= override still wins — a deliberate local choice, not silent enrolment, and the documented way to exercise a canary with telemetry off. The CLI check mirrors shouldTrack() rather than importing it: client.ts already imports canary.ts for canaryEventProperties, so depending on it would be a cycle. Both read the same two inputs, so they cannot disagree. Tests: 9 new across CLI and Studio (preference off, each runtime override, no bucket assigned, override still honoured, flag properties all-false). Fault injection: removing the CLI gate fails 6, removing the Studio gate fails 3. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 9f2b892 commit 4f464dc

6 files changed

Lines changed: 186 additions & 33 deletions

File tree

‎docs/contributing/canary-rollouts.mdx‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -220,6 +220,20 @@ rm -rf ~/.hyperframes # clears telemetry id, canary cohorts, and breaker stat
220220
`hyperframes telemetry status` prints both paths if you want to inspect or
221221
delete them individually. Nothing canary-related is stored anywhere else.
222222

223+
**Opting out of telemetry opts you out of canaries.** A canary is a *measured*
224+
rollout — we enrol a slice precisely so it can be compared against everyone
225+
else. An install that sends nothing can't be compared, so enrolling it buys no
226+
signal and only changes that user's code path, on an experimental feature,
227+
without their knowledge. Every opt-out route counts: the persisted preference
228+
(`hyperframes telemetry disable`), the runtime env vars
229+
(`HYPERFRAMES_NO_TELEMETRY`, `DO_NOT_TRACK`), dev/telemetry-disabled builds,
230+
and Studio's `hyperframes-studio:telemetryDisabled`. The decision resolves to
231+
`telemetry_opt_out` *before* bucketing, so no cohort is assigned at all.
232+
233+
An explicit `HF_CANARY_<FEATURE>` (or `?hf_canary_<feature>=` in Studio)
234+
override still wins — that is a deliberate local choice, and it stays the way
235+
to exercise a canary with telemetry off.
236+
223237
**It fails closed.** No install id, an unregistered name, or a CI machine all
224238
resolve to *not enrolled*. A canary exists to bound blast radius, so "we
225239
don't know who this is" must never mean "enrol everyone". CI is excluded

‎packages/cli/src/telemetry/canary.test.ts‎

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,32 @@
11
import { describe, expect, it, vi, beforeEach } from "vitest";
22

3-
const configState: { anonymousId: string; bucketSeed: string | undefined } = {
3+
const configState: {
4+
anonymousId: string;
5+
bucketSeed: string | undefined;
6+
telemetryEnabled: boolean;
7+
} = {
48
anonymousId: "db0c1f4a-b95e-4c35-90c6-1a15bd76f717",
59
bucketSeed: undefined,
10+
telemetryEnabled: true,
611
};
712
const systemState = { is_ci: false };
13+
// null = no runtime opt-out in force; a string names the source (env var,
14+
// dev build, ...) exactly as policy.ts reports it.
15+
const policyState: { runtimeOverride: string | null } = { runtimeOverride: null };
816

917
vi.mock("./config.js", () => ({
10-
readConfig: () => ({ anonymousId: configState.anonymousId, bucketSeed: configState.bucketSeed }),
18+
readConfig: () => ({
19+
anonymousId: configState.anonymousId,
20+
bucketSeed: configState.bucketSeed,
21+
telemetryEnabled: configState.telemetryEnabled,
22+
}),
1123
}));
1224
vi.mock("./system.js", () => ({
1325
getSystemMeta: () => ({ is_ci: systemState.is_ci }),
1426
}));
27+
vi.mock("./policy.js", () => ({
28+
telemetryRuntimeOverride: () => policyState.runtimeOverride,
29+
}));
1530

1631
// The registry is data; pin a known shape so these tests don't move when a
1732
// real canary is added or ramped.
@@ -64,11 +79,53 @@ beforeEach(() => {
6479
__resetCanaryCacheForTests();
6580
configState.anonymousId = "db0c1f4a-b95e-4c35-90c6-1a15bd76f717";
6681
configState.bucketSeed = undefined;
82+
configState.telemetryEnabled = true;
6783
systemState.is_ci = false;
84+
policyState.runtimeOverride = null;
6885
delete process.env.HF_CANARY_TEST_ALPHA;
6986
delete process.env.HF_CANARY_TEST_BETA;
7087
});
7188

89+
describe("telemetry opt-out is canary opt-out", () => {
90+
it("does not enrol when the persisted preference is off", () => {
91+
configState.telemetryEnabled = false;
92+
// test-alpha is at 100% — it would be on for everyone otherwise.
93+
expect(resolveCanary("test-alpha")).toEqual({
94+
enabled: false,
95+
reason: "telemetry_opt_out",
96+
});
97+
});
98+
99+
it.each(["HYPERFRAMES_NO_TELEMETRY", "DO_NOT_TRACK", "dev_mode"])(
100+
"does not enrol under the %s runtime override, even with the preference on",
101+
(source) => {
102+
configState.telemetryEnabled = true;
103+
policyState.runtimeOverride = source;
104+
expect(resolveCanary("test-alpha").reason).toBe("telemetry_opt_out");
105+
},
106+
);
107+
108+
it("never buckets an opted-out install — no cohort is assigned at all", () => {
109+
configState.telemetryEnabled = false;
110+
// A bucket number would mean we hashed them into a slice anyway.
111+
expect(resolveCanary("test-alpha").bucket).toBeUndefined();
112+
});
113+
114+
it("still honours an explicit override — the documented way to test with telemetry off", () => {
115+
configState.telemetryEnabled = false;
116+
process.env.HF_CANARY_TEST_BETA = "on";
117+
expect(resolveCanary("test-beta")).toEqual({ enabled: true, reason: "forced_on" });
118+
});
119+
120+
it("reports every canary as false to PostHog shape when opted out", () => {
121+
configState.telemetryEnabled = false;
122+
expect(canaryEventProperties()).toEqual({
123+
"$feature/canary-test-alpha": "false",
124+
"$feature/canary-test-beta": "false",
125+
});
126+
});
127+
});
128+
72129
describe("bucketing unit", () => {
73130
it("buckets on the bucketSeed when present — the unit that survives config wipes", async () => {
74131
const { evaluateCanary } = await import("@hyperframes/core/canary");

‎packages/cli/src/telemetry/canary.ts‎

Lines changed: 55 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,28 @@ import {
2828
} from "@hyperframes/core/canary";
2929
import { CANARIES, canaryEnvVar, findCanary } from "@hyperframes/core/canary-registry";
3030
import { readConfig } from "./config.js";
31+
import { telemetryRuntimeOverride } from "./policy.js";
3132
import { getSystemMeta } from "./system.js";
3233

34+
/**
35+
* Opting out of telemetry opts you out of canaries.
36+
*
37+
* A canary is a measured rollout: we enrol a slice precisely so we can compare
38+
* it against everyone else. An install that sends nothing can't be compared,
39+
* so enrolling it buys no signal — it only changes that user's code path, on
40+
* an experimental feature, without their knowledge and with no way for us to
41+
* see the result. That is the wrong side of an opt-out.
42+
*
43+
* Deliberately mirrors `shouldTrack()` rather than importing it: client.ts
44+
* already imports this module for `canaryEventProperties`, so depending on it
45+
* here would be a cycle. Both read the same two inputs (`policy.ts`'s runtime
46+
* override, then the persisted preference), so they cannot disagree.
47+
*/
48+
function telemetryActive(): boolean {
49+
if (telemetryRuntimeOverride() !== null) return false;
50+
return readConfig().telemetryEnabled;
51+
}
52+
3353
/**
3454
* Decisions are memoized per process: a `--batch` run asks the same question
3555
* once per row, and a canary must not change its mind mid-process — a render
@@ -43,6 +63,40 @@ export function __resetCanaryCacheForTests(): void {
4363
decisions.clear();
4464
}
4565

66+
/** The uncached decision. Split out so `resolveCanary` is purely the memo. */
67+
function decideCanary(name: string): CanaryDecision {
68+
const definition = findCanary(name);
69+
if (!definition) return { enabled: false, reason: "out_of_cohort" };
70+
71+
const override = parseCanaryOverride(process.env[canaryEnvVar(definition.name)]);
72+
73+
// Resolved ahead of evaluate so an opted-out install is never bucketed at
74+
// all. An explicit HF_CANARY_* override still wins: that is a deliberate
75+
// local choice (support session, bisect, developer testing), not silent
76+
// enrolment, and it stays the way to exercise a canary with telemetry off.
77+
if (override === undefined && !telemetryActive()) {
78+
return { enabled: false, reason: "telemetry_opt_out" };
79+
}
80+
81+
const config = readConfig();
82+
return evaluateCanary({
83+
feature: definition.name,
84+
// The bucket seed, NOT the anonymousId: the seed is inherited across
85+
// config.json re-mints via the install-state file, so cohorts hold when
86+
// the telemetry id re-rolls. Both files live in ~/.hyperframes, so
87+
// deleting that directory clears the cohort too — intentional. Fallback
88+
// covers only a failed backfill write on a legacy config.
89+
unitId: config.bucketSeed ?? config.anonymousId,
90+
percentage: definition.percentage,
91+
override,
92+
// CI installs regenerate their config per run, so their ids are
93+
// ephemeral — they would hop cohorts between runs, adding noise to the
94+
// rollout signal while saying nothing about real users. An explicit
95+
// override still gets through, which is how you test a canary in CI.
96+
exclude: getSystemMeta().is_ci,
97+
});
98+
}
99+
46100
/**
47101
* Full decision for a registered canary, including the reason — use this when
48102
* you want to record WHY, not just whether.
@@ -54,27 +108,7 @@ export function resolveCanary(name: string): CanaryDecision {
54108
const cached = decisions.get(name);
55109
if (cached) return cached;
56110

57-
const definition = findCanary(name);
58-
const config = readConfig();
59-
const decision: CanaryDecision = definition
60-
? evaluateCanary({
61-
feature: definition.name,
62-
// The bucket seed, NOT the anonymousId: the seed is inherited across
63-
// config.json re-mints via the install-state file, so cohorts hold
64-
// when the telemetry id re-rolls. Both files live in ~/.hyperframes,
65-
// so deleting that directory clears the cohort too — intentional.
66-
// Fallback covers only a failed backfill write on a legacy config.
67-
unitId: config.bucketSeed ?? config.anonymousId,
68-
percentage: definition.percentage,
69-
override: parseCanaryOverride(process.env[canaryEnvVar(definition.name)]),
70-
// CI installs regenerate their config per run, so their ids are
71-
// ephemeral — they would hop cohorts between runs, adding noise to the
72-
// rollout signal while saying nothing about real users. An explicit
73-
// override still gets through, which is how you test a canary in CI.
74-
exclude: getSystemMeta().is_ci,
75-
})
76-
: { enabled: false, reason: "out_of_cohort" };
77-
111+
const decision = decideCanary(name);
78112
decisions.set(name, decision);
79113
return decision;
80114
}

‎packages/core/src/canary.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,13 @@ export type CanaryReason =
3939
| "in_cohort"
4040
| "out_of_cohort"
4141
| "no_unit_id"
42-
| "excluded";
42+
| "excluded"
43+
// Telemetry is off, so the install is not enrolled. Distinct from
44+
// "excluded" because the caller decides this BEFORE evaluate is reached —
45+
// and because "why is my canary off" has a very different answer in the two
46+
// cases. Never appears in telemetry by construction: an install that
47+
// resolves this way sends nothing.
48+
| "telemetry_opt_out";
4349

4450
export interface CanaryDecision {
4551
enabled: boolean;

‎packages/studio/src/telemetry/canary.test.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,3 +192,36 @@ describe("telemetry", () => {
192192
expect(canaryEventProperties()["$feature/canary-on-everywhere"]).toBe("false");
193193
});
194194
});
195+
196+
describe("telemetry opt-out is canary opt-out", () => {
197+
// The studio opt-out lever, per telemetry/config.ts.
198+
const OPT_OUT_KEY = "hyperframes-studio:telemetryDisabled";
199+
200+
it("does not enrol an opted-out browser profile", () => {
201+
localStorage.setItem(OPT_OUT_KEY, "1");
202+
// on-everywhere is at 100% — it would be on for everyone otherwise.
203+
expect(resolveCanary("on-everywhere")).toEqual({
204+
enabled: false,
205+
reason: "telemetry_opt_out",
206+
});
207+
});
208+
209+
it("never buckets an opted-out profile — no cohort is assigned at all", () => {
210+
localStorage.setItem(OPT_OUT_KEY, "1");
211+
expect(resolveCanary("on-everywhere").bucket).toBeUndefined();
212+
});
213+
214+
it("still honours an explicit URL override", () => {
215+
localStorage.setItem(OPT_OUT_KEY, "1");
216+
setSearch("?hf_canary_off_everywhere=on");
217+
expect(resolveCanary("off-everywhere")).toEqual({ enabled: true, reason: "forced_on" });
218+
});
219+
220+
it("reports every canary as false when opted out", () => {
221+
localStorage.setItem(OPT_OUT_KEY, "1");
222+
expect(canaryEventProperties()).toEqual({
223+
"$feature/canary-on-everywhere": "false",
224+
"$feature/canary-off-everywhere": "false",
225+
});
226+
});
227+
});

‎packages/studio/src/telemetry/canary.ts‎

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ import {
4141
} from "@hyperframes/core/canary";
4242
import { CANARIES, findCanary } from "@hyperframes/core/canary-registry";
4343
import { resolveStudioDistinctId } from "./distinctId";
44+
import { isOptedOut } from "./config";
4445
import { safeSessionStorage } from "../utils/safeStorage";
4546

4647
/** `my-feature` → `hf_canary_my_feature`, the query param and storage key. */
@@ -151,15 +152,23 @@ export function resolveCanary(name: string): CanaryDecision {
151152
if (cached) return cached;
152153

153154
const definition = findCanary(name);
154-
const decision: CanaryDecision = definition
155-
? evaluateCanary({
156-
feature: definition.name,
157-
unitId: resolveBucketUnit(),
158-
percentage: definition.percentage,
159-
override: readOverride(definition.name),
160-
exclude: isAutomatedBrowser(),
161-
})
162-
: { enabled: false, reason: "out_of_cohort" };
155+
const override = definition ? readOverride(definition.name) : undefined;
156+
// Opting out of telemetry opts you out of canaries — same rule as the CLI
157+
// (see packages/cli/src/telemetry/canary.ts). An install that sends nothing
158+
// can't be compared against anyone, so enrolling it changes that user's
159+
// code path for no signal. An explicit override still wins: that is a
160+
// deliberate local choice, not silent enrolment.
161+
const decision: CanaryDecision = !definition
162+
? { enabled: false, reason: "out_of_cohort" }
163+
: override === undefined && isOptedOut()
164+
? { enabled: false, reason: "telemetry_opt_out" }
165+
: evaluateCanary({
166+
feature: definition.name,
167+
unitId: resolveBucketUnit(),
168+
percentage: definition.percentage,
169+
override,
170+
exclude: isAutomatedBrowser(),
171+
});
163172

164173
decisions.set(name, decision);
165174
return decision;

0 commit comments

Comments
 (0)