Skip to content

Commit b3990ac

Browse files
vanceingallsclaude
andcommitted
feat(studio): emit the canary reason on Studio events too
Review caught that the same anti-pattern was still live in the Studio binding: canaryEventProperties destructured only `enabled` and dropped the reason. Its own doc comment promised 'identical shape to the CLI, so a rollout spanning both reads as one flag' — which the CLI-only fix had just made false. This matters beyond symmetry. A CLI-launched Studio adopts the CLI's decisions and shares its bucket seed, so a cohort flip can surface on either surface. Emitting attribution on only one leaves Studio-observed flips unattributable and makes the two flip counts irreconcilable — and Studio is the surface most likely to expose a shared-seed-with-diverging-id pattern, which is the open question the reason exists to answer. Also adds the no_unit_id emission test the CLI side advertised but never asserted, and a Studio pair pinning that a URL override and a cohort roll produce the same assignment with different reasons. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 076657a commit b3990ac

3 files changed

Lines changed: 47 additions & 4 deletions

File tree

‎packages/cli/src/telemetry/canary.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,14 @@ describe("reason reaches the event properties", () => {
267267
expect(canaryEventProperties()["canary_reason_test_gamma"]).toBe("excluded");
268268
});
269269

270+
// The PR body advertises no_unit_id as one of two values that pay for
271+
// themselves; the resolver exercises it but the emission layer did not.
272+
it("reports no_unit_id at the emission layer when the install has no id", () => {
273+
configState.anonymousId = "";
274+
configState.bucketSeed = undefined;
275+
expect(canaryEventProperties()["canary_reason_test_gamma"]).toBe("no_unit_id");
276+
});
277+
270278
it("reports telemetry_opt_out when the preference is off", () => {
271279
configState.telemetryEnabled = false;
272280
// Emitted for completeness; by construction such an install sends nothing,

‎packages/studio/src/telemetry/canary.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,8 @@ describe("telemetry", () => {
194194
expect(canaryEventProperties()).toEqual({
195195
"$feature/canary-on-everywhere": "true",
196196
"$feature/canary-off-everywhere": "false",
197+
canary_reason_on_everywhere: "in_cohort",
198+
canary_reason_off_everywhere: "out_of_cohort",
197199
});
198200

199201
__resetStudioCanaryCacheForTests();
@@ -235,6 +237,8 @@ describe("telemetry opt-out is canary opt-out", () => {
235237
expect(canaryEventProperties()).toEqual({
236238
"$feature/canary-on-everywhere": "false",
237239
"$feature/canary-off-everywhere": "false",
240+
canary_reason_on_everywhere: "telemetry_opt_out",
241+
canary_reason_off_everywhere: "telemetry_opt_out",
238242
});
239243
});
240244
});
@@ -323,3 +327,24 @@ describe("CLI-launched Studio adopts the CLI's decisions", () => {
323327
});
324328
});
325329
});
330+
331+
// A CLI-launched Studio shares the CLI's bucket seed, so a cohort flip can
332+
// surface on either surface. Attribution on only one of them makes the two
333+
// flip counts irreconcilable — which is why this is not a CLI-only property.
334+
describe("Studio attribution matches the CLI", () => {
335+
it("distinguishes a local URL override from a cohort roll at the same value", () => {
336+
setSearch("?hf_canary_off_everywhere=on");
337+
const props = canaryEventProperties();
338+
expect(props["$feature/canary-off-everywhere"]).toBe("true");
339+
expect(props["canary_reason_off_everywhere"]).toBe("forced_on");
340+
// Same assignment `on-everywhere` reaches by an ordinary roll.
341+
expect(props["$feature/canary-on-everywhere"]).toBe("true");
342+
expect(props["canary_reason_on_everywhere"]).toBe("in_cohort");
343+
});
344+
345+
it("never puts a reason inside the $feature namespace", () => {
346+
for (const [key, value] of Object.entries(canaryEventProperties())) {
347+
if (key.startsWith("$feature/")) expect(value).toMatch(/^(true|false)$/);
348+
}
349+
});
350+
});

‎packages/studio/src/telemetry/canary.ts‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -257,12 +257,22 @@ export function isCanaryEnabled(name: string): boolean {
257257
}
258258

259259
/**
260-
* Canary assignments as PostHog flag properties (`$feature/canary-<name>`),
261-
* attached to every Studio event so any metric can be split by cohort —
262-
* identical shape to the CLI, so a rollout spanning both reads as one flag.
260+
* Canary assignments as PostHog flag properties (`$feature/canary-<name>`)
261+
* plus their `canary_reason_<name>`, attached to every Studio event so any
262+
* metric can be split by cohort — identical shape to the CLI, so a rollout
263+
* spanning both reads as one flag.
264+
*
265+
* The reason has to be here too, not just CLI-side. A CLI-launched Studio
266+
* adopts the CLI's decisions and shares its bucket seed, so a cohort flip can
267+
* surface on either surface; emitting attribution on only one of them makes
268+
* the two flip counts irreconcilable and leaves Studio-observed flips
269+
* unattributable — the exact ambiguity this property exists to remove.
263270
*/
264271
export function canaryEventProperties(): Record<string, string> {
265272
return canaryFeatureProperties(
266-
CANARIES.map((c) => ({ name: c.name, enabled: resolveCanary(c.name).enabled })),
273+
CANARIES.map((c) => {
274+
const { enabled, reason } = resolveCanary(c.name);
275+
return { name: c.name, enabled, reason };
276+
}),
267277
);
268278
}

0 commit comments

Comments
 (0)