Skip to content

Commit a2f2827

Browse files
vanceingallsclaude
andcommitted
feat(cli): bucket canaries on a machine-lineage seed, not the telemetry id
Cohort membership now survives a config wipe. Canaries bucket on a dedicated bucketSeed (fresh random UUID, distinct from anonymousId by design) that is mirrored write-once into the install-state file and inherited at mint: a wipe re-rolls the telemetry id but never the canary assignment. This removes cumulative-exposure drift for the recoverable churn bucket entirely — the residual drift comes only from fresh machines, containers, and genuinely new users — and keeps before/after comparisons valid across a reinstall. The seed is never emitted in telemetry (only the resulting true/false assignments are), so it does not link the old id to the new one server-side. The residual linker is the flag vector itself (k bits for k live canaries), documented as such. An explicit reset still works by deleting the state file, and the no-identity test now also asserts the seed differs from the anonymousId. Cross-surface coherence: the CLI's studio server injects the seed as window.__HF_CLI_BUCKET_SEED (same telemetry gate and script-escaping as the distinct id, and on the /api/telemetry-identity fallback), and the Studio binding buckets on it when present — without this the CLI would bucket on the seed while Studio bucketed on the distinct id, splitting one machine across cohorts (calibration check 4 would catch exactly this). Standalone Studio still buckets on its localStorage id: the browser has no second storage location, so that id doubles as the seed. Legacy configs are backfilled once (lineage seed if the state file has one, else minted) and persisted immediately — an unpersisted seed would re-roll cohorts every process. Safe to ship in the same release as the first canaries: no prior release emitted canary properties, so the bucketing-unit change is unobservable. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent ef1df3a commit a2f2827

10 files changed

Lines changed: 327 additions & 53 deletions

File tree

‎docs/contributing/canary-rollouts.mdx‎

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -169,30 +169,41 @@ experiment, and it survives uninstall), and account identity covers only
169169
accepted limit — but its worst consequence is mitigated, and its size is now
170170
directly measurable rather than inferred:
171171

172+
- **Cohorts survive the wipe.** Canaries bucket on a dedicated `bucketSeed` —
173+
not the telemetry id — and the seed is inherited across config wipes via a
174+
machine-local state file (write-once: the first install's seed is the
175+
lineage's seed forever). A wipe re-rolls the telemetry id, never the canary
176+
assignment. The seed is never emitted, so it does not link the old id to
177+
the new one.
172178
- **A re-rolled install cannot re-enter a path that already failed on that
173-
machine.** The circuit breaker's tripped state is mirrored to the
174-
machine-local state file, so it survives the same config wipe that re-rolls
175-
the cohort.
179+
machine.** The circuit breaker's tripped state is mirrored to the same
180+
state file.
176181
- **`install_predecessor_found`** on every event says whether this install's
177182
mint found a previous install's state marker. Its true-share splits check
178183
2's drift into the recoverable part (config wiped, machine persisted) and
179184
the part no local mechanism can link (fresh machine, container, new user).
180185

181-
The calibration read is therefore two numbers — total drift, and the fraction
182-
the rollover already covers — and canary window lengths get picked from the
183-
residual, not the raw turnover.
186+
With the seed carryover, check 2's residual drift comes from the
187+
unrecoverable buckets only — fresh machines, containers, and genuinely new
188+
users — and canary window lengths get picked from that residual, not the raw
189+
turnover.
184190

185191
## Behaviour worth knowing
186192

187193
**Ramping is inclusive.** Widening `10 → 25` keeps everyone who was already
188194
in the 10. Cohorts never reshuffle, so a before/after comparison stays valid
189195
across a ramp.
190196

191-
**Slices are independent per feature.** The bucket hashes `feature:installId`,
192-
not the install id alone — two canaries at 10% select two different 10%s. If
197+
**Slices are independent per feature.** The bucket hashes `feature:seed`,
198+
not the seed alone — two canaries at 10% select two different 10%s. If
193199
they shared a slice, one unlucky cohort would receive every experiment at
194200
once and no two rollouts could be read apart.
195201

202+
**Cohorts are keyed to the machine, not the config.** The bucketing unit is a
203+
dedicated seed inherited across config wipes (see the calibration section) —
204+
distinct from the telemetry id, never emitted, and shared with a CLI-launched
205+
Studio so both surfaces agree.
206+
196207
**It fails closed.** No install id, an unregistered name, or a CI machine all
197208
resolve to *not enrolled*. A canary exists to bound blast radius, so "we
198209
don't know who this is" must never mean "enrol everyone". CI is excluded

‎packages/cli/src/server/studioServer.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,11 @@ import {
1616
loadRuntimeSourceSignature,
1717
} from "./runtimeSource.js";
1818
import { VERSION as version } from "../version.js";
19-
import { buildStudioHeadScripts, resolveCliTelemetryDistinctId } from "./telemetryIdentity.js";
19+
import {
20+
buildStudioHeadScripts,
21+
resolveCliBucketSeed,
22+
resolveCliTelemetryDistinctId,
23+
} from "./telemetryIdentity.js";
2024
import { emitStudioRenderComplete, emitStudioRenderError } from "./studioRenderTelemetry.js";
2125
import { isDevMode } from "../utils/env.js";
2226
import {
@@ -652,7 +656,10 @@ export function createStudioServer(options: StudioServerOptions): StudioServer {
652656
// distinct id (no PII) so the browser session can join the CLI's PostHog
653657
// person, or `{ distinctId: null }` when CLI telemetry is disabled.
654658
app.get("/api/telemetry-identity", (c) => {
655-
return c.json({ distinctId: resolveCliTelemetryDistinctId() });
659+
return c.json({
660+
distinctId: resolveCliTelemetryDistinctId(),
661+
bucketSeed: resolveCliBucketSeed(),
662+
});
656663
});
657664

658665
app.get("/api/events", (c) => {

‎packages/cli/src/server/telemetryIdentity.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,14 @@ describe("buildCliIdentityScript", () => {
6666
);
6767
});
6868

69+
it("also seeds window.__HF_CLI_BUCKET_SEED when the config carries a bucket seed", () => {
70+
shouldTrack.mockReturnValue(true);
71+
readConfig.mockReturnValue({ anonymousId: "machine-uuid", bucketSeed: "seed-uuid" });
72+
expect(buildCliIdentityScript()).toBe(
73+
'<script>window.__HF_CLI_DISTINCT_ID="machine-uuid";window.__HF_CLI_BUCKET_SEED="seed-uuid";</script>',
74+
);
75+
});
76+
6977
it("emits an empty string when telemetry is disabled (nothing to seed)", () => {
7078
shouldTrack.mockReturnValue(false);
7179
expect(buildCliIdentityScript()).toBe("");

‎packages/cli/src/server/telemetryIdentity.ts‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,15 +41,39 @@ export function resolveCliTelemetryDistinctId(): string | null {
4141
* `url_hash` telemetry or browser history. Empty string when there's nothing to
4242
* seed (telemetry off / no id).
4343
*/
44+
/**
45+
* The CLI's canary bucket seed to hand to Studio, or null. Injected alongside
46+
* the distinct id so a CLI-launched Studio buckets canaries on the SAME unit
47+
* as the CLI — without it the two surfaces would agree only while the seed
48+
* still equals whatever Studio falls back to, and a rollout spanning render
49+
* and editor would split one user across cohorts. Same telemetry gate as the
50+
* distinct id: seeding is part of the identity stitch, not a separate channel.
51+
*/
52+
export function resolveCliBucketSeed(): string | null {
53+
try {
54+
if (!telemetryShouldTrack()) return null;
55+
const seed = readConfig().bucketSeed;
56+
return typeof seed === "string" && seed.length > 0 ? seed : null;
57+
} catch {
58+
return null;
59+
}
60+
}
61+
62+
// JSON.stringify does not escape "<" or "/". Escaping both means no
63+
// "</script>" (or "</…") sequence can form in the emitted value, so it can
64+
// never terminate the inline <script> or open a new tag. (The values are
65+
// randomUUID()s, so this is belt-and-suspenders.)
66+
function encodeInlineScriptValue(value: string): string {
67+
return JSON.stringify(value).replace(/</g, "\\u003c").replace(/\//g, "\\/");
68+
}
69+
4470
export function buildCliIdentityScript(): string {
4571
const cliId = resolveCliTelemetryDistinctId();
4672
if (!cliId) return "";
47-
// The id is a randomUUID() so this is belt-and-suspenders, but JSON.stringify
48-
// does not escape "<" or "/". Escaping both means no "</script>" (or "</…")
49-
// sequence can form in the emitted value, so it can never terminate the
50-
// inline <script> or open a new tag.
51-
const encoded = JSON.stringify(cliId).replace(/</g, "\\u003c").replace(/\//g, "\\/");
52-
return `<script>window.__HF_CLI_DISTINCT_ID=${encoded};</script>`;
73+
const parts = [`window.__HF_CLI_DISTINCT_ID=${encodeInlineScriptValue(cliId)};`];
74+
const seed = resolveCliBucketSeed();
75+
if (seed) parts.push(`window.__HF_CLI_BUCKET_SEED=${encodeInlineScriptValue(seed)};`);
76+
return `<script>${parts.join("")}</script>`;
5377
}
5478

5579
/**

‎packages/cli/src/telemetry/canary.test.ts‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
import { describe, expect, it, vi, beforeEach } from "vitest";
22

3-
const configState = { anonymousId: "db0c1f4a-b95e-4c35-90c6-1a15bd76f717" };
3+
const configState: { anonymousId: string; bucketSeed: string | undefined } = {
4+
anonymousId: "db0c1f4a-b95e-4c35-90c6-1a15bd76f717",
5+
bucketSeed: undefined,
6+
};
47
const systemState = { is_ci: false };
58

69
vi.mock("./config.js", () => ({
7-
readConfig: () => ({ anonymousId: configState.anonymousId }),
10+
readConfig: () => ({ anonymousId: configState.anonymousId, bucketSeed: configState.bucketSeed }),
811
}));
912
vi.mock("./system.js", () => ({
1013
getSystemMeta: () => ({ is_ci: systemState.is_ci }),
@@ -60,11 +63,44 @@ const { isCanaryEnabled, resolveCanary, canaryEventProperties, __resetCanaryCach
6063
beforeEach(() => {
6164
__resetCanaryCacheForTests();
6265
configState.anonymousId = "db0c1f4a-b95e-4c35-90c6-1a15bd76f717";
66+
configState.bucketSeed = undefined;
6367
systemState.is_ci = false;
6468
delete process.env.HF_CANARY_TEST_ALPHA;
6569
delete process.env.HF_CANARY_TEST_BETA;
6670
});
6771

72+
describe("bucketing unit", () => {
73+
it("buckets on the bucketSeed when present — the unit that survives config wipes", async () => {
74+
const { evaluateCanary } = await import("@hyperframes/core/canary");
75+
configState.bucketSeed = "5f1c9d2e-0000-4000-8000-aaaaaaaaaaaa";
76+
const viaBinding = resolveCanary("test-alpha").bucket;
77+
const bySeed = evaluateCanary({
78+
feature: "test-alpha",
79+
unitId: configState.bucketSeed,
80+
percentage: 100,
81+
}).bucket;
82+
const byId = evaluateCanary({
83+
feature: "test-alpha",
84+
unitId: configState.anonymousId,
85+
percentage: 100,
86+
}).bucket;
87+
expect(viaBinding).toBe(bySeed);
88+
// Only meaningful if the two units actually bucket differently.
89+
expect(bySeed).not.toBe(byId);
90+
});
91+
92+
it("falls back to the anonymousId when no seed exists (failed legacy backfill)", async () => {
93+
const { evaluateCanary } = await import("@hyperframes/core/canary");
94+
const viaBinding = resolveCanary("test-alpha").bucket;
95+
const byId = evaluateCanary({
96+
feature: "test-alpha",
97+
unitId: configState.anonymousId,
98+
percentage: 100,
99+
}).bucket;
100+
expect(viaBinding).toBe(byId);
101+
});
102+
});
103+
68104
describe("CLI canary binding", () => {
69105
it("reads the percentage from the registry", () => {
70106
expect(isCanaryEnabled("test-alpha")).toBe(true);

‎packages/cli/src/telemetry/canary.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,15 @@ export function resolveCanary(name: string): CanaryDecision {
5555
if (cached) return cached;
5656

5757
const definition = findCanary(name);
58+
const config = readConfig();
5859
const decision: CanaryDecision = definition
5960
? evaluateCanary({
6061
feature: definition.name,
61-
unitId: readConfig().anonymousId,
62+
// The bucket seed, NOT the anonymousId: the seed is inherited across
63+
// config wipes via the install-state file, so the machine keeps its
64+
// cohorts when the telemetry id re-rolls. Fallback covers only a
65+
// failed backfill write on a legacy config.
66+
unitId: config.bucketSeed ?? config.anonymousId,
6267
percentage: definition.percentage,
6368
override: parseCanaryOverride(process.env[canaryEnvVar(definition.name)]),
6469
// CI installs regenerate their config per run, so their ids are

‎packages/cli/src/telemetry/config.test.ts‎

Lines changed: 76 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -180,11 +180,18 @@ describe("install-state rollover (breaker survives a config wipe)", () => {
180180
expect(readConfigFresh().deParallelRouterTrialFired).toBe(true);
181181
});
182182

183-
it("the state file holds no identity — only the marker timestamp and breaker fact", () => {
183+
it("the state file holds no telemetry identity — marker, breaker fact, bucket seed only", () => {
184184
const config = readConfig();
185185
config.deParallelRouterTrialFired = true;
186186
writeConfig(config);
187-
expect(Object.keys(stateFile()).sort()).toEqual(["deParallelRouterTrialFired", "markerAt"]);
187+
expect(Object.keys(stateFile()).sort()).toEqual([
188+
"bucketSeed",
189+
"deParallelRouterTrialFired",
190+
"markerAt",
191+
]);
192+
// The seed is a separate random UUID, never the anonymousId — the old
193+
// telemetry id must not survive a wipe in any form.
194+
expect(config.bucketSeed).not.toBe(config.anonymousId);
188195
expect(JSON.stringify(stateFile())).not.toContain(config.anonymousId);
189196
});
190197

@@ -233,3 +240,70 @@ describe("install-state rollover (breaker survives a config wipe)", () => {
233240
expect(readConfigFresh().predecessorFound).toBeUndefined();
234241
});
235242
});
243+
244+
describe("bucket-seed carryover (cohorts survive a config wipe)", () => {
245+
let readConfig: typeof import("./config.js").readConfig;
246+
let readConfigFresh: typeof import("./config.js").readConfigFresh;
247+
let writeConfig: typeof import("./config.js").writeConfig;
248+
let CONFIG_PATH: typeof import("./config.js").CONFIG_PATH;
249+
let STATE_PATH: typeof import("./config.js").STATE_PATH;
250+
251+
beforeEach(async () => {
252+
fsState.files.clear();
253+
vi.resetModules();
254+
({ readConfig, readConfigFresh, writeConfig, CONFIG_PATH, STATE_PATH } =
255+
await import("./config.js"));
256+
});
257+
258+
it("a fresh install mints a seed distinct from its anonymousId and mirrors it to the state file", () => {
259+
const config = readConfig();
260+
expect(config.bucketSeed).toBeTruthy();
261+
expect(config.bucketSeed).not.toBe(config.anonymousId);
262+
const state = JSON.parse(fsState.files.get(STATE_PATH) as string) as { bucketSeed?: string };
263+
expect(state.bucketSeed).toBe(config.bucketSeed);
264+
});
265+
266+
it("the seed survives a config wipe — the machine keeps its cohorts, only the id re-rolls", () => {
267+
const first = readConfig();
268+
fsState.files.delete(CONFIG_PATH);
269+
const second = readConfigFresh();
270+
expect(second.bucketSeed).toBe(first.bucketSeed);
271+
expect(second.anonymousId).not.toBe(first.anonymousId);
272+
});
273+
274+
it("the seed survives config corruption via the same mint path", () => {
275+
const first = readConfig();
276+
fsState.files.set(CONFIG_PATH, "{not valid json");
277+
expect(readConfigFresh().bucketSeed).toBe(first.bucketSeed);
278+
});
279+
280+
it("the state-file seed is write-once — a later install never overwrites the lineage seed", () => {
281+
const first = readConfig();
282+
fsState.files.delete(CONFIG_PATH);
283+
const second = readConfigFresh();
284+
// Force more config writes from the second install; the state seed must not move.
285+
second.commandCount = 7;
286+
writeConfig(second);
287+
const state = JSON.parse(fsState.files.get(STATE_PATH) as string) as { bucketSeed?: string };
288+
expect(state.bucketSeed).toBe(first.bucketSeed);
289+
});
290+
291+
it("a legacy config without a seed is backfilled ONCE and stays stable across fresh reads", () => {
292+
const base = readConfig();
293+
const { bucketSeed: _dropped, ...legacy } = base;
294+
fsState.files.set(CONFIG_PATH, JSON.stringify(legacy));
295+
fsState.files.delete(STATE_PATH); // no lineage either — pure legacy machine
296+
const first = readConfigFresh();
297+
expect(first.bucketSeed).toBeTruthy();
298+
const second = readConfigFresh();
299+
expect(second.bucketSeed).toBe(first.bucketSeed); // persisted, not re-rolled per read
300+
});
301+
302+
it("a legacy config adopts the lineage seed from the state file when one exists", () => {
303+
const base = readConfig(); // wrote the state file with a seed
304+
const lineageSeed = base.bucketSeed;
305+
const { bucketSeed: _dropped, ...legacy } = base;
306+
fsState.files.set(CONFIG_PATH, JSON.stringify(legacy));
307+
expect(readConfigFresh().bucketSeed).toBe(lineageSeed);
308+
});
309+
});

0 commit comments

Comments
 (0)