Skip to content

Commit 3b65321

Browse files
feat(cli): classify identity persistence on every telemetry event (#3065)
* feat(cli): classify identity persistence on every telemetry event Install-grain metrics currently trust every anonymousId equally, but ephemeral/isolated-HOME workloads mint a fresh id per run — one machine produced 2,956 rotating render identities since Jul 30 (94.4% seen on a single render command), inflating acquisition and diluting per-install penetration while looking like real product usage. Every event now carries: - identity_persistence: durable (id loaded from a preexisting config — proven to survive a process boundary) | unknown (minted+persisted this run; an ephemeral HOME is indistinguishable from a genuine first run from inside one process) | process_only (persist failed). Sticky per process so a fresh install re-reading its own write cannot self-promote. - config_write_outcome: ok | ok_unmirrored | failed for the identity- establishing write; absent when the id came from disk. - invocation_id: random uuid per CLI process, so one invocation's events group even when the install identity is untrustworthy (unlike run_id, which needs an orchestrator to set HYPERFRAMES_RUN_ID). Install metrics can then count only durable identities, and a daily churn monitor can alert on the unknown share. * fix(cli): require the anonymousId to come off disk before classifying durable Review finding: materializeConfig mints a replacement anonymousId when a hand-edited/image-baked config lacks one. That replacement only reaches disk when the bucket-seed backfill happens to write; with a seed present the read path performs no write at all, so the install re-mints a fresh id every run while the unconditional durable branch stamped each of them with the one label durable-only counting is allowed to trust. durable now requires parseNonEmptyString(parsed.anonymousId): a minted replacement classifies like a fresh mint — by the backfill write outcome when that path runs (unknown/process_only), and process_only on the no-write path where the id provably dies with the process. Two tests pin both shapes.
1 parent 88853f1 commit 3b65321

6 files changed

Lines changed: 209 additions & 4 deletions

File tree

‎packages/cli/src/telemetry/client.postureRefresh.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ const configState = { telemetryEnabled: true };
1616
vi.mock("./config.js", () => ({
1717
readConfig: () => ({ anonymousId: "anon-1", telemetryEnabled: configState.telemetryEnabled }),
1818
writeConfig: () => {},
19+
getIdentityPersistence: () => "durable",
20+
getIdentityWriteOutcome: () => undefined,
1921
}));
2022
vi.mock("../utils/env.js", () => ({ isDevMode: () => false }));
2123
vi.mock("./canary.js", () => ({ canaryEventProperties: () => ({}) }));

‎packages/cli/src/telemetry/client.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ vi.stubEnv("DO_NOT_TRACK", "");
1010
vi.mock("./config.js", () => ({
1111
readConfig: () => ({ anonymousId: "anon-test-123", telemetryEnabled: true }),
1212
writeConfig: () => {},
13+
getIdentityPersistence: () => "durable",
14+
getIdentityWriteOutcome: () => undefined,
1315
}));
1416

1517
// shouldTrack() short-circuits in dev mode — force production behavior.

‎packages/cli/src/telemetry/client.ts‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1-
import { readConfig, writeConfig } from "./config.js";
1+
import {
2+
getIdentityPersistence,
3+
getIdentityWriteOutcome,
4+
readConfig,
5+
writeConfig,
6+
} from "./config.js";
7+
import { getInvocationId } from "./runId.js";
28
import { VERSION } from "../version.js";
39
import { c } from "../ui/colors.js";
410
import { diag } from "../ui/diagnostics.js";
@@ -101,6 +107,19 @@ export function trackEvent(
101107
// could not read. Without it a partial disk write is indistinguishable
102108
// from a genuinely fresh install. Absent in the normal case.
103109
install_state_file_corrupt: readConfig().stateFileCorrupt,
110+
// Whether this process's anonymousId can be trusted to survive to the
111+
// next run: `durable` (loaded from a preexisting config), `unknown`
112+
// (minted+persisted this run — an ephemeral HOME is indistinguishable
113+
// from a genuine first run), `process_only` (persist failed). Install-
114+
// grain metrics should count only durable identities; the identity-
115+
// churn workloads (fresh id per run) are never durable.
116+
identity_persistence: getIdentityPersistence(),
117+
// Outcome of the identity-establishing config write; absent when the
118+
// identity came from disk and nothing needed writing.
119+
config_write_outcome: getIdentityWriteOutcome(),
120+
// Groups one invocation's events even when the install identity is
121+
// untrustworthy. Always present, unlike the orchestrator-set run_id.
122+
invocation_id: getInvocationId(),
104123
// Canary assignments as `$feature/canary-<name>` — PostHog's native flag
105124
// property shape, so breakdowns and experiment analysis work on a canary
106125
// with nothing configured server-side. On EVERY event, not just renders:

‎packages/cli/src/telemetry/config.test.ts‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -699,3 +699,92 @@ describe("an unwritable config dir must not re-roll the seed", () => {
699699
});
700700
});
701701
});
702+
703+
describe("identity-persistence classification (sticky per process)", () => {
704+
let readConfig: typeof import("./config.js").readConfig;
705+
let readConfigFresh: typeof import("./config.js").readConfigFresh;
706+
let getIdentityPersistence: typeof import("./config.js").getIdentityPersistence;
707+
let getIdentityWriteOutcome: typeof import("./config.js").getIdentityWriteOutcome;
708+
let CONFIG_PATH: typeof import("./config.js").CONFIG_PATH;
709+
710+
beforeEach(async () => {
711+
fsState.files.clear();
712+
policyState.runtimeOverride = null;
713+
vi.resetModules();
714+
({ readConfig, readConfigFresh, getIdentityPersistence, getIdentityWriteOutcome, CONFIG_PATH } =
715+
await import("./config.js"));
716+
});
717+
718+
it("classifies a fresh mint whose write landed as unknown, never durable", () => {
719+
readConfig();
720+
expect(getIdentityPersistence()).toBe("unknown");
721+
expect(getIdentityWriteOutcome()).toBe("ok");
722+
});
723+
724+
it("classifies an id loaded from a preexisting config as durable, with no write outcome", () => {
725+
fsState.files.set(
726+
CONFIG_PATH,
727+
JSON.stringify({ telemetryEnabled: true, anonymousId: "prior-id", bucketSeed: "seed" }),
728+
);
729+
readConfig();
730+
expect(getIdentityPersistence()).toBe("durable");
731+
expect(getIdentityWriteOutcome()).toBeUndefined();
732+
});
733+
734+
it("classifies a fresh mint whose write failed as process_only", async () => {
735+
const fs = await import("node:fs");
736+
vi.mocked(fs.writeFileSync).mockImplementation(() => {
737+
throw new Error("EACCES: permission denied");
738+
});
739+
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
740+
741+
readConfig();
742+
expect(getIdentityPersistence()).toBe("process_only");
743+
expect(getIdentityWriteOutcome()).toBe("failed");
744+
745+
warn.mockRestore();
746+
vi.mocked(fs.writeFileSync).mockImplementation((path, content) => {
747+
fsState.files.set(String(path), String(content));
748+
});
749+
});
750+
751+
it("does not self-promote to durable when a fresh-install process re-reads its own write", () => {
752+
readConfig();
753+
expect(fsState.files.has(CONFIG_PATH)).toBe(true);
754+
// The file now exists on disk; a cache-bypassing re-read hits the
755+
// existing-file path — the ephemeral-HOME churn signature.
756+
readConfigFresh();
757+
expect(getIdentityPersistence()).toBe("unknown");
758+
});
759+
760+
it("does not label a replacement id minted at read time as durable (seed present, no write)", () => {
761+
// Hand-edited / image-baked config: file exists with a seed but NO
762+
// anonymousId. materializeConfig mints a replacement, nothing persists it
763+
// (the seed suppresses the backfill write) — so the install re-mints
764+
// every run. Labelling that durable would dress the churn signature in
765+
// the one trustworthy label (review finding).
766+
fsState.files.set(
767+
CONFIG_PATH,
768+
JSON.stringify({ telemetryEnabled: true, bucketSeed: "baked-seed" }),
769+
);
770+
readConfig();
771+
expect(getIdentityPersistence()).toBe("process_only");
772+
});
773+
774+
it("classifies a replacement id carried to disk by the seed backfill by its write outcome", () => {
775+
// Same hand-edited shape but ALSO missing the seed: the backfill write
776+
// persists the whole config, replacement id included — a fresh mint in
777+
// all but name, so it classifies like one (unknown, never durable).
778+
fsState.files.set(CONFIG_PATH, JSON.stringify({ telemetryEnabled: true }));
779+
readConfig();
780+
expect(getIdentityPersistence()).toBe("unknown");
781+
expect(getIdentityWriteOutcome()).toBe("ok");
782+
});
783+
784+
it("classifies a corrupt-config recovery mint by its write outcome, not as durable", () => {
785+
fsState.files.set(CONFIG_PATH, "{not json");
786+
readConfig();
787+
expect(getIdentityPersistence()).toBe("unknown");
788+
expect(getIdentityWriteOutcome()).toBe("ok");
789+
});
790+
});

‎packages/cli/src/telemetry/config.ts‎

Lines changed: 78 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -147,11 +147,12 @@ function warnSeedBackfillFailed(error: string | undefined): void {
147147
* is exactly when that happens, so it says so once rather than failing
148148
* invisibly.
149149
*/
150-
function backfillBucketSeed(config: HyperframesConfig): void {
150+
function backfillBucketSeed(config: HyperframesConfig): ConfigWriteResult {
151151
const recorded = readInstallState();
152152
config.bucketSeed = (isInstallState(recorded) ? recorded.bucketSeed : undefined) ?? randomUUID();
153153
const write = writeConfigWithResult(config);
154154
if (!write.ok) warnSeedBackfillFailed(write.error);
155+
return write;
155156
}
156157

157158
// ONLY the positive is cached. The latch is monotonic across processes in one
@@ -350,6 +351,7 @@ function mintAndCacheConfig(): HyperframesConfig {
350351
const config = mintConfig();
351352
const write = writeConfigWithResult(config);
352353
if (!write.ok) warnSeedBackfillFailed(write.error);
354+
classifyIdentity(write.ok ? "unknown" : "process_only", writeOutcomeOf(write));
353355
cachedConfig = { ...config };
354356
return { ...config };
355357
}
@@ -535,6 +537,60 @@ const DEFAULT_CONFIG: HyperframesConfig = {
535537

536538
let cachedConfig: HyperframesConfig | null = null;
537539

540+
// ---------------------------------------------------------------------------
541+
// Identity-persistence classification — one sticky verdict per process.
542+
//
543+
// Install-grain metrics need to know whether this process's anonymousId can
544+
// be trusted to survive to the next run. Three-way, because from inside a
545+
// single process durability is not always provable:
546+
//
547+
// durable — the id was LOADED from a preexisting config file: it has
548+
// already survived at least one process boundary.
549+
// unknown — the id was minted this run and the write landed. An
550+
// ephemeral/isolated HOME (the identity-churn workloads:
551+
// fresh id per run, install_predecessor_found=false every
552+
// time) looks IDENTICAL to a genuine first run from in
553+
// here, so this cannot be promoted to durable.
554+
// process_only — minted this run and the write failed (read-only mount,
555+
// full disk): the id dies with this process, guaranteed.
556+
//
557+
// The verdict is sticky: a fresh-install process that later re-reads its own
558+
// just-written file must not upgrade itself to durable.
559+
// ---------------------------------------------------------------------------
560+
561+
export type IdentityPersistence = "durable" | "process_only" | "unknown";
562+
/** `ok_unmirrored`: config.json landed but the install-state mirror did not. */
563+
export type IdentityWriteOutcome = "ok" | "ok_unmirrored" | "failed";
564+
565+
let identityPersistence: IdentityPersistence | undefined;
566+
let identityWriteOutcome: IdentityWriteOutcome | undefined;
567+
568+
function classifyIdentity(persistence: IdentityPersistence, outcome?: IdentityWriteOutcome): void {
569+
if (identityPersistence !== undefined) return;
570+
identityPersistence = persistence;
571+
identityWriteOutcome = outcome;
572+
}
573+
574+
function writeOutcomeOf(write: ConfigWriteResult): IdentityWriteOutcome {
575+
if (!write.ok) return "failed";
576+
return write.mirrored === false ? "ok_unmirrored" : "ok";
577+
}
578+
579+
/** The process's sticky identity-persistence verdict (classifies on demand). */
580+
export function getIdentityPersistence(): IdentityPersistence {
581+
readConfig();
582+
return identityPersistence ?? "unknown";
583+
}
584+
585+
/**
586+
* Outcome of the identity-establishing config write. Absent when the identity
587+
* came from disk and nothing needed writing (the `durable` case).
588+
*/
589+
export function getIdentityWriteOutcome(): IdentityWriteOutcome | undefined {
590+
readConfig();
591+
return identityWriteOutcome;
592+
}
593+
538594
/** A non-empty string, or undefined — hand-edited configs can carry anything. */
539595
function parseNonEmptyString(value: unknown): string | undefined {
540596
return typeof value === "string" && value.length > 0 ? value : undefined;
@@ -637,18 +693,36 @@ export function readConfig(): HyperframesConfig {
637693

638694
const config = materializeConfig(parsed);
639695

696+
// `durable` requires the id to have actually COME OFF DISK — a file that
697+
// predates this process proves cross-run persistence. materializeConfig
698+
// mints a REPLACEMENT id when the parsed file lacks one (hand-edited /
699+
// image-baked configs), and that replacement only reaches disk if the
700+
// bucket-seed backfill below happens to write; labelling it durable would
701+
// dress the exact churn signature this field exists to catch in the one
702+
// trustworthy label (review finding). Sticky either way, so a
703+
// fresh-install process re-reading its own write cannot self-promote.
704+
const idFromDisk = parseNonEmptyString(parsed.anonymousId) !== undefined;
705+
640706
// One-time backfill for configs predating the bucket seed: prefer the
641707
// recorded seed if a previous install already wrote one, else mint.
642708
// Persisted immediately — an unpersisted seed would re-roll every process.
643709
if (config.bucketSeed === undefined) {
644-
backfillBucketSeed(config);
710+
const write = backfillBucketSeed(config);
711+
// The backfill write carries any replacement id to disk, so a minted id
712+
// classifies exactly like a fresh mint: by whether the write landed.
713+
if (idFromDisk) classifyIdentity("durable");
714+
else classifyIdentity(write.ok ? "unknown" : "process_only", writeOutcomeOf(write));
645715
// Cache even if the write failed, so the seed is at least stable for
646716
// the life of this process (a re-roll per readConfigFresh would flip
647717
// cohorts mid-session).
648718
cachedConfig = config;
649719
return { ...config };
650720
}
651721

722+
// No write happens on this path: a replacement id lives only in this
723+
// process, guaranteed — the definition of process_only.
724+
classifyIdentity(idFromDisk ? "durable" : "process_only");
725+
652726
cachedConfig = config;
653727
return { ...config };
654728
} catch {
@@ -658,7 +732,8 @@ export function readConfig(): HyperframesConfig {
658732
// breaker survives config corruption too — but fail closed for the
659733
// privacy control: recovery must never silently turn telemetry back on.
660734
const config = { ...mintConfig(), telemetryEnabled: false };
661-
writeConfig(config);
735+
const write = writeConfigWithResult(config);
736+
classifyIdentity(write.ok ? "unknown" : "process_only", writeOutcomeOf(write));
662737
return config;
663738
}
664739
}

‎packages/cli/src/telemetry/runId.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import { randomUUID } from "node:crypto";
2+
13
let resolved = false;
24
let runId: string | undefined;
35

@@ -10,3 +12,19 @@ export function getRunId(): string | undefined {
1012

1113
return runId;
1214
}
15+
16+
// ---------------------------------------------------------------------------
17+
// Invocation id — a random uuid minted once per CLI process, present on every
18+
// event that process emits. Unlike run_id (set only when an orchestrator
19+
// exports HYPERFRAMES_RUN_ID), it needs no environment plumbing: it exists so
20+
// the events of ONE invocation can be grouped even when the install identity
21+
// is untrustworthy (identity_persistence != durable, e.g. an ephemeral HOME
22+
// minting a fresh anonymousId per run).
23+
// ---------------------------------------------------------------------------
24+
25+
let invocationId: string | undefined;
26+
27+
export function getInvocationId(): string {
28+
invocationId ??= randomUUID();
29+
return invocationId;
30+
}

0 commit comments

Comments
 (0)