Skip to content

Commit 9f028bf

Browse files
committed
feat(studio): instrument the audio FX rack, including work an agent did
Nothing in the FX feature reported anything, so every question about it — which presets get used, whether the named jobs beat a bare peaking filter, whether anybody touches the one-knob profiles — was unanswerable. **The events.** `audio_fx_*` through the existing `trackStudioEvent` transport, defined in one module (`audioFxTelemetry.ts`) rather than scattered across six components, because the shapes have to agree for a dashboard to join them. Presets applied/removed/amount/automated/ auditioned; nodes added (with `via: job|effect|eq`), removed, bypassed, moved; parameters committed (with `surface: knob|details`); profile knobs; carve changes; the leveller; the Tone EQ. `track_kind` rides on all of them, so every answer can be read per material. Two rules the module states and enforces: - **Nothing user-authored leaves the browser.** No element ids, no filenames, no chain JSON, no parameter values in the provenance payload — only catalogue identifiers, counts and booleans. Tested. - **Commit, not preview.** Every rack control has a continuous preview path and a commit path; wiring the preview would emit tens of events per drag and describe mouse travel rather than decisions. The shelf's hover-audition is deduped to once per preset per shelf-open for the same reason. **Agent attribution**, which was the harder half. Two tiers: 1. The CLI already fingerprints twelve agent runtimes from its own env, and Studio cannot see any of it. So `buildCliIdentityScript` now publishes the resolved category as `window.__HF_CLI_AGENT_RUNTIME`, and both studio session-property builders attach it to EVERY event — not just audio ones, since "person or agent" is worth asking of any feature. Gated on telemetry being on, unlike the canary decisions: those exist so Studio does not enrol, this only labels an event, and with telemetry off there is no event to label. 2. An agent asked to fix a mix does not drive the panel — it edits the composition HTML or runs `scripts/carve.mjs`, and none of the events above will ever fire. So the panel watches the chain's own shape and reports `audio_fx_chain_observed` when it changes, carrying how many edits this session had made. A chain that moved while that counter stood still moved because something outside the studio moved it. The write path is wrapped once rather than counted at ten callsites — the count is only meaningful if it is exhaustive. Encoding "none" rather than omitting the property, on both transports: most sessions are people, that is a finding, and this project's telemetry has already produced one wrong conclusion from comparing a populated sentinel against an absence. Not done, deliberately: `carve.mjs` is not instrumented — its output is already identifiable by the `fromCarve` tag on the nodes it writes, so tier 2 covers it without giving a script context its own policy, id and transport to get wrong.
1 parent 7443ac2 commit 9f028bf

12 files changed

Lines changed: 781 additions & 20 deletions

‎packages/cli/src/server/telemetryIdentity.test.ts‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,11 @@ const canaryDecisions = vi.fn<() => Record<string, { enabled: boolean; forced: b
1717
// assertion below ran against a refresh that silently did nothing.
1818
const resetPostureCache = vi.fn();
1919
const readConfigFresh = vi.fn();
20+
// Mocked for the same reason as the canary registry above: the real detector
21+
// reads THIS process env, so every string assertion here would pass on a
22+
// developer laptop and fail the moment an agent ran the suite (CLAUDECODE=1),
23+
// or vice versa. The agent-specific behaviour gets its own cases below.
24+
const detectAgent = vi.fn<() => string | null>();
2025

2126
vi.mock("../telemetry/client.js", () => ({
2227
shouldTrack: (...args: unknown[]) => shouldTrack(...args),
@@ -26,6 +31,9 @@ vi.mock("../telemetry/config.js", () => ({
2631
readConfig: (...args: unknown[]) => readConfig(...args),
2732
readConfigFresh: () => readConfigFresh(),
2833
}));
34+
vi.mock("../telemetry/agent_runtime.js", () => ({
35+
detectAgentRuntime: () => detectAgent(),
36+
}));
2937
vi.mock("../telemetry/canary.js", () => ({
3038
canaryDecisionsForStudio: () => canaryDecisions(),
3139
}));
@@ -46,6 +54,10 @@ describe("resolveCliTelemetryDistinctId", () => {
4654
readConfig.mockReset();
4755
canaryDecisions.mockReset();
4856
canaryDecisions.mockReturnValue({});
57+
detectAgent.mockReset();
58+
// No agent is the default so the existing assertions keep describing the
59+
// ordinary case: a person at a terminal.
60+
detectAgent.mockReturnValue(null);
4961
});
5062

5163
it("returns the CLI anonymousId when telemetry is enabled", () => {
@@ -393,3 +405,61 @@ describe("cross-process opt-out refresh", () => {
393405
expect(after).not.toContain("__HF_CLI_BUCKET_SEED");
394406
});
395407
});
408+
409+
/**
410+
* Publishing which agent, if any, drives the CLI.
411+
*
412+
* Studio has no way to detect this — the signal is in the CLI process
413+
* environment, which the browser never sees — so this injection is the only
414+
* path by which an agent-driven session can ever be labelled as one.
415+
*/
416+
describe("buildCliIdentityScript agent runtime", () => {
417+
beforeEach(() => {
418+
shouldTrack.mockReset();
419+
readConfig.mockReset();
420+
readConfig.mockReturnValue({});
421+
canaryDecisions.mockReset();
422+
canaryDecisions.mockReturnValue({});
423+
detectAgent.mockReset();
424+
detectAgent.mockReturnValue(null);
425+
});
426+
427+
it("publishes the agent when one is driving the CLI", () => {
428+
shouldTrack.mockReturnValue(true);
429+
detectAgent.mockReturnValue("claude_code");
430+
expect(buildCliIdentityScript()).toContain('window.__HF_CLI_AGENT_RUNTIME="claude_code";');
431+
});
432+
433+
it("publishes nothing when a person is driving it", () => {
434+
shouldTrack.mockReturnValue(true);
435+
expect(buildCliIdentityScript()).not.toContain("__HF_CLI_AGENT_RUNTIME");
436+
});
437+
438+
it("stays silent when telemetry is off, even under an agent", () => {
439+
// Unlike the canary decisions, which Studio needs in order NOT to enrol,
440+
// this is only ever read to label an event. With telemetry off there is no
441+
// event, so publishing it would leave a marker in the page of someone who
442+
// asked not to be measured.
443+
shouldTrack.mockReturnValue(false);
444+
detectAgent.mockReturnValue("claude_code");
445+
expect(buildCliIdentityScript()).not.toContain("__HF_CLI_AGENT_RUNTIME");
446+
});
447+
448+
it("publishes it without an identity when the Host is not trusted", () => {
449+
// The value is a category, not an id — a LAN Studio should still be able to
450+
// say an agent opened it, the same way it still receives canary decisions.
451+
shouldTrack.mockReturnValue(true);
452+
detectAgent.mockReturnValue("codex");
453+
const script = buildCliIdentityScript({ includeIdentity: false });
454+
expect(script).toContain('window.__HF_CLI_AGENT_RUNTIME="codex";');
455+
expect(script).not.toContain("__HF_CLI_DISTINCT_ID");
456+
});
457+
458+
it("escapes a value that tries to close the script tag", () => {
459+
shouldTrack.mockReturnValue(true);
460+
detectAgent.mockReturnValue("</script><script>alert(1)</script>");
461+
const script = buildCliIdentityScript();
462+
expect(script).not.toContain("</script><script>");
463+
expect(script.match(/<\/script>/g)).toHaveLength(1);
464+
});
465+
});

‎packages/cli/src/server/telemetryIdentity.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import {
2323
shouldTrack as telemetryShouldTrack,
2424
} from "../telemetry/client.js";
2525
import { canaryDecisionsForStudio, type CliCanaryDecision } from "../telemetry/canary.js";
26+
import { detectAgentRuntime } from "../telemetry/agent_runtime.js";
2627

2728
/**
2829
* The CLI's anonymous distinct id to hand to Studio, or null when CLI telemetry
@@ -156,6 +157,27 @@ export function buildCliIdentityScript(options: { includeIdentity?: boolean } =
156157
parts.push(`window.__HF_CLI_CANARY_DECISIONS=${encodeInlineScriptJson(decisions)};`);
157158
}
158159

160+
// Which agent, if any, is driving this CLI — and therefore the Studio it just
161+
// opened. Studio cannot work this out for itself: the signal is entirely in
162+
// the CLI process's environment, which the browser never sees.
163+
//
164+
// Published on the same terms as the decisions above, and for the same
165+
// reason: it is a category derived from the EXISTENCE of well-known vendor
166+
// env vars, never their values (`agent_runtime.ts` is explicit that it never
167+
// reads a value, because some are API keys). One of a dozen fixed strings, or
168+
// absent. Nothing identifying, so it does not belong behind the trusted-Host
169+
// gate that gates the distinct id.
170+
//
171+
// Gated on telemetry being ON, unlike the decisions above. Those exist so a
172+
// Studio whose CLI opted out does not enrol itself, so they have to survive
173+
// the opt-out. This is only ever read to label an event, so with telemetry off
174+
// there is nothing for it to label — publishing it anyway would leave a marker
175+
// in the page of a user who asked not to be measured.
176+
const agent = telemetryShouldTrack() ? detectAgentRuntime() : null;
177+
if (agent) {
178+
parts.push(`window.__HF_CLI_AGENT_RUNTIME=${encodeInlineScriptValue(agent)};`);
179+
}
180+
159181
return parts.length === 0 ? "" : `<script>${parts.join("")}</script>`;
160182
}
161183

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
// @vitest-environment happy-dom
2+
3+
import { describe, expect, it, vi, beforeEach } from "vitest";
4+
import type { HfAudioFxChain } from "@hyperframes/core/audio-fx";
5+
6+
const trackStudioEvent = vi.fn();
7+
vi.mock("../../utils/studioTelemetry", () => ({
8+
trackStudioEvent: (...args: unknown[]) => trackStudioEvent(...args),
9+
}));
10+
11+
const { chainShape, nodeOrigin, trackChainObserved, trackParamCommitted, trackPresetApplied } =
12+
await import("./audioFxTelemetry");
13+
14+
beforeEach(() => trackStudioEvent.mockReset());
15+
16+
const chainOf = (nodes: HfAudioFxChain["nodes"]): HfAudioFxChain => ({ version: 1, nodes });
17+
18+
describe("what the rack reports", () => {
19+
it("namespaces every event so the feature can be found as one thing", () => {
20+
trackPresetApplied("voice-clean", "voice", 5, "append", { trackKind: "voice" });
21+
expect(trackStudioEvent).toHaveBeenCalledWith("audio_fx_preset_applied", expect.anything());
22+
});
23+
24+
it("carries what the track is, so a preset's use can be read per material", () => {
25+
trackPresetApplied("telephone", "character", 7, "append", { trackKind: "music" });
26+
expect(trackStudioEvent.mock.calls[0]?.[1]).toMatchObject({
27+
preset: "telephone",
28+
family: "character",
29+
track_kind: "music",
30+
});
31+
});
32+
33+
it("says 'unknown' rather than nothing when the track is unclassified", () => {
34+
// Absent and "unclassified" are different facts; encoding them the same way
35+
// is what makes a breakdown read as a gap that is not there.
36+
trackPresetApplied("hall", "space", 1, "append", {});
37+
expect(trackStudioEvent.mock.calls[0]?.[1]).toMatchObject({ track_kind: "unknown" });
38+
});
39+
40+
it("separates the two surfaces a parameter can be set from", () => {
41+
// The whole one-knob design rests on which of these people use.
42+
trackParamCommitted("compressor", "ratio", 4, "details", {});
43+
expect(trackStudioEvent.mock.calls[0]?.[1]).toMatchObject({
44+
effect: "compressor",
45+
param: "ratio",
46+
value: 4,
47+
surface: "details",
48+
});
49+
});
50+
});
51+
52+
describe("a chain reduced to counts", () => {
53+
it("names where each node came from", () => {
54+
expect(nodeOrigin({ type: "peaking", fromPreset: "voice-clean" })).toBe("preset");
55+
expect(nodeOrigin({ type: "peaking", fromCarve: true })).toBe("carve");
56+
expect(nodeOrigin({ type: "gain", fromLeveller: true })).toBe("leveller");
57+
expect(nodeOrigin({ type: "highpass" })).toBe("hand");
58+
});
59+
60+
it("counts by origin, which is what separates hand-built from generated", () => {
61+
const shape = chainShape(
62+
chainOf([
63+
{ type: "highpass", fromPreset: "voice-clean" },
64+
{ type: "peaking", fromPreset: "voice-clean" },
65+
{ type: "peaking", fromCarve: true },
66+
{ type: "lowpass" },
67+
]),
68+
);
69+
expect(shape).toMatchObject({
70+
node_count: 4,
71+
nodes_from_preset: 2,
72+
nodes_from_carve: 1,
73+
nodes_by_hand: 1,
74+
preset_count: 1,
75+
});
76+
});
77+
78+
it("counts distinct presets, not preset nodes — stacking is the question", () => {
79+
const shape = chainShape(
80+
chainOf([
81+
{ type: "highpass", fromPreset: "voice-clean" },
82+
{ type: "lowpass", fromPreset: "telephone" },
83+
{ type: "peaking", fromPreset: "telephone" },
84+
]),
85+
);
86+
expect(shape.preset_count).toBe(2);
87+
});
88+
89+
it("survives an empty or absent chain", () => {
90+
expect(chainShape(null)).toMatchObject({ node_count: 0, preset_count: 0 });
91+
});
92+
93+
it("ships no chain contents — only counts and flags", () => {
94+
// The standing rule: nothing user-authored leaves the browser. A parameter
95+
// value or an element id in here would be the author's own data.
96+
const shape = chainShape(chainOf([{ type: "peaking", id: "n1", params: { frequency: 3000 } }]));
97+
const serialized = JSON.stringify(shape);
98+
expect(serialized).not.toContain("3000");
99+
expect(serialized).not.toContain("n1");
100+
});
101+
});
102+
103+
describe("effects this session did not apply", () => {
104+
it("calls a chain with no panel edits behind it authored outside", () => {
105+
// This is the agent signal. An agent edits the composition and the panel
106+
// simply finds the work done — none of its own events ever fire.
107+
trackChainObserved(
108+
chainOf([{ type: "highpass", fromPreset: "voice-clean" }]),
109+
{ firstSight: true, panelEdits: 0, hasCarve: false, hasAutomation: false },
110+
{ trackKind: "voice" },
111+
);
112+
expect(trackStudioEvent).toHaveBeenCalledWith(
113+
"audio_fx_chain_observed",
114+
expect.objectContaining({ authored_outside: true, first_sight: true }),
115+
);
116+
});
117+
118+
it("does not, when this session's own edits explain it", () => {
119+
trackChainObserved(
120+
chainOf([{ type: "highpass" }]),
121+
{ firstSight: false, panelEdits: 3, hasCarve: false, hasAutomation: false },
122+
{},
123+
);
124+
expect(trackStudioEvent.mock.calls[0]?.[1]).toMatchObject({
125+
authored_outside: false,
126+
panel_edits: 3,
127+
});
128+
});
129+
});

0 commit comments

Comments
 (0)