Skip to content

Commit f822200

Browse files
feat(telemetry): measure which lint rules fire, cost, and fail to converge (#3367)
* feat(telemetry): measure which lint rules fire, cost, and fail to converge Lint rule changes are currently argued from anecdote. This adds the three measurements needed to argue them from data. `lint_report`, once per `hyperframes lint` or `hyperframes check`: - `code_counts` / `codes` — which rules actually fire, and how often - `rule_group_ms` — milliseconds per rule-source module (core, gsap, media, ...) - `slowest_rule` / `slowest_rule_ms` — slowest single rule as `<group>#<index>` - `rule_count` — how many rules this build ran `lint_rule_streak`, once per finding that survives an edit to its file: - `edits` — how many edits the finding survived - `cleared` — whether it eventually went away The streak event is the one that matters. A lint pass costs about 5ms, so per-rule CPU is not what makes the authoring loop slow; a rule an agent cannot satisfy is, because every failed attempt costs a full edit-and-relint cycle. A single run cannot see that, so `lint_rule_streak` reconstructs it across runs: high `edits` with `cleared: false` is a rule nobody can fix, and the `cleared: true` distribution is the baseline to judge it against. An iteration is counted only when the file's content digest CHANGED and the finding is still there. Re-linting an untouched project is not an attempt, which is what stops `check` (which lints on every invocation) from inflating the numbers. Rule identity is the source module plus an index within it. Naming all 86 rules would make the timings prettier but it is a refactor this measurement does not need: the group locates the file, and the index locates the rule. Version, agent runtime, CI flag, and invocation id are already attached to every event by `trackEvent`, so lint pain can be split by CLI version and by which agent produced it without adding anything here. Privacy: only rule codes, counts, and timings are sent. Streak state lives in ~/.hyperframes/lint-streaks.json alongside config.json (so `rm -rf ~/.hyperframes` is still a full reset) and stores digests only — no file paths, no project names, no composition source. Nothing is written and nothing is emitted when telemetry is off. Entries expire after 14 days and are capped at 500 files. `EventProperties` gains string arrays and numeric maps. `codes` and `code_counts` are inherently a set and a histogram; flattening them into dynamic top-level keys would make them unqueryable. PostHog stores both natively. `trackLintRun` is the single call site shared by `lint` and `check`, and it swallows every error — telemetry must never turn a green lint red. * feat(telemetry): emit per-group rule counts so slowest_rule stays comparable Review catch on #3367: `slowest_rule` is the one positional key in either event. It is `<group>#<index>`, so adding or removing a rule renumbers every later slot in that group and the same string means different rules in two builds. #3366 does exactly that to 34 of 81 surviving slots, and `rule_count` alone says only THAT the ruleset moved, not which groups. `rule_group_counts` carries the per-group sizes alongside it, so a consumer comparing two builds can tell which groups' indices still mean the same thing without anyone having to remember which release dropped rules. `codes`, `code_counts` and `rule_group_ms` are keyed by name and were never affected. Also corrects the rule count in the RULE_GROUPS comment: 86, not ~60, as LINT_RULE_COUNT in the same file computes.
1 parent 83ceaeb commit f822200

15 files changed

Lines changed: 640 additions & 34 deletions

‎packages/cli/src/commands/check.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ function cleanLint(): ProjectLintResult {
5858
results: [
5959
{
6060
file: "index.html",
61+
contentHash: "test",
6162
result: {
6263
ok: true,
6364
errorCount: 0,
@@ -82,6 +83,7 @@ function lintWith(
8283
results: [
8384
{
8485
file: "index.html",
86+
contentHash: "test",
8587
result: {
8688
ok: severity !== "error",
8789
errorCount: severity === "error" ? 1 : 0,

‎packages/cli/src/commands/lint.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ export const examples: Example[] = [
1212
import { formatLintFindings } from "../utils/lintFormat.js";
1313
import { lintProject } from "../utils/lintProject.js";
1414
import { resolveProject } from "../utils/project.js";
15+
import { trackLintRun } from "../telemetry/lintRun.js";
16+
import { getRunId } from "../telemetry/runId.js";
1517
import { withMeta } from "../utils/updateCheck.js";
1618

1719
export default defineCommand({
@@ -45,7 +47,13 @@ export default defineCommand({
4547
// (publish/transcribe/upgrade/play/present) already use.
4648
try {
4749
const project = resolveProject(args.dir);
50+
const startedAt = Date.now();
4851
const lintResult = await lintProject(project.dir);
52+
trackLintRun(project.dir, lintResult, {
53+
command: "lint",
54+
durationMs: Date.now() - startedAt,
55+
...(getRunId() !== undefined ? { runId: getRunId() } : {}),
56+
});
4957

5058
if (args.json) {
5159
const allFindings = lintResult.results.flatMap((r) => r.result.findings);

‎packages/cli/src/commands/validate.test.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,7 @@ describe("extractCompositionErrorsFromLint", () => {
232232
results: [
233233
{
234234
file: "index.html",
235+
contentHash: "test",
235236
result: {
236237
ok: findings.length === 0,
237238
errorCount: 0,
@@ -282,6 +283,7 @@ describe("extractCompositionErrorsFromLint", () => {
282283
results: [
283284
{
284285
file: "index.html",
286+
contentHash: "test",
285287
result: {
286288
ok: false,
287289
errorCount: 1,
@@ -298,6 +300,7 @@ describe("extractCompositionErrorsFromLint", () => {
298300
},
299301
{
300302
file: "compositions/nested.html",
303+
contentHash: "test",
301304
result: {
302305
ok: false,
303306
errorCount: 1,

‎packages/cli/src/telemetry/events.ts‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -819,3 +819,76 @@ export function trackCheckReport(props: {
819819
...runIdField(props.runId),
820820
});
821821
}
822+
823+
/**
824+
* One lint pass over a project. `code_counts` is what makes "which rules
825+
* actually fire" answerable; `rule_group_ms` and `slowest_rule` are what make
826+
* "which rules are expensive" answerable. Only lint rule codes and timings are
827+
* sent — never file paths, project names, or composition source.
828+
*/
829+
export function trackLintReport(props: {
830+
/** The command that ran the lint: "lint" or "check". */
831+
command: string;
832+
durationMs: number;
833+
filesScanned: number;
834+
errorCount: number;
835+
warningCount: number;
836+
infoCount: number;
837+
/** Finding count keyed by lint rule code. */
838+
codeCounts: Record<string, number>;
839+
/** Milliseconds spent per rule-source module, summed across files. */
840+
ruleGroupMs: Record<string, number>;
841+
/** Slowest single rule as `<group>#<index>`, across every file in the run. */
842+
slowestRule: string;
843+
slowestRuleMs: number;
844+
/** How many rules this build ran, so a ruleset change is visible in the data. */
845+
ruleCount: number;
846+
/**
847+
* Rule count per group. `slowest_rule` is positional, so a group that changed
848+
* size between two builds has indices that no longer mean the same thing.
849+
*/
850+
ruleGroupCounts: Record<string, number>;
851+
runId?: string;
852+
}): void {
853+
trackEvent("lint_report", {
854+
command: props.command,
855+
duration_ms: Math.round(props.durationMs),
856+
files_scanned: props.filesScanned,
857+
error_count: props.errorCount,
858+
warning_count: props.warningCount,
859+
info_count: props.infoCount,
860+
codes: Object.keys(props.codeCounts).sort(),
861+
code_counts: props.codeCounts,
862+
rule_group_ms: props.ruleGroupMs,
863+
slowest_rule: props.slowestRule,
864+
slowest_rule_ms: Math.round(props.slowestRuleMs),
865+
rule_count: props.ruleCount,
866+
rule_group_counts: props.ruleGroupCounts,
867+
...runIdField(props.runId),
868+
});
869+
}
870+
871+
/**
872+
* A finding that survived one or more edits to the file it was reported on.
873+
*
874+
* `cleared: false` with a high `edits` is the signal that matters most: a rule
875+
* an agent kept trying and failing to satisfy. `cleared: true` gives the
876+
* distribution to compare it against — how many edits a normal finding costs.
877+
*/
878+
export function trackLintRuleStreak(props: {
879+
code: string;
880+
severity: string;
881+
edits: number;
882+
cleared: boolean;
883+
command: string;
884+
runId?: string;
885+
}): void {
886+
trackEvent("lint_rule_streak", {
887+
code: props.code,
888+
severity: props.severity,
889+
edits: props.edits,
890+
cleared: props.cleared,
891+
command: props.command,
892+
...runIdField(props.runId),
893+
});
894+
}
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
// End-to-end: a real project on disk -> lintProject -> the exact PostHog
2+
// payloads. Unit tests cover the streak arithmetic; this proves the wiring
3+
// and pins the event shape a dashboard will be built against.
4+
5+
import { describe, it, expect, vi, beforeEach } from "vitest";
6+
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
7+
import { tmpdir } from "node:os";
8+
import { join } from "node:path";
9+
10+
const HOME = mkdtempSync(join(tmpdir(), "hf-lintrun-"));
11+
vi.mock("node:os", async (importOriginal) => {
12+
const actual = await importOriginal<typeof import("node:os")>();
13+
return { ...actual, homedir: () => HOME };
14+
});
15+
16+
// Capture at the transport boundary so everything client.ts adds
17+
// (cli_version, invocation_id, ...) is visible in the assertions.
18+
const enqueued: Array<{ event: string; properties: Record<string, unknown> }> = [];
19+
vi.mock("./transport.js", () => ({
20+
enqueue: (event: string, properties: Record<string, unknown>) =>
21+
enqueued.push({ event, properties }),
22+
flush: () => Promise.resolve(),
23+
}));
24+
// shouldTrack() consults these two. A dev build disables telemetry by default,
25+
// which would make this test assert on an empty queue and pass for the wrong
26+
// reason.
27+
vi.mock("./policy.js", async (importOriginal) => {
28+
const actual = await importOriginal<typeof import("./policy.js")>();
29+
return { ...actual, telemetryRuntimeOverride: () => null };
30+
});
31+
vi.mock("./config.js", async (importOriginal) => {
32+
const actual = await importOriginal<typeof import("./config.js")>();
33+
return { ...actual, readConfig: () => ({ ...actual.readConfig(), telemetryEnabled: true }) };
34+
});
35+
36+
const { trackLintRun } = await import("./lintRun.js");
37+
const { lintProject } = await import("@hyperframes/lint");
38+
39+
const COMPOSITION = `<html><body>
40+
<div id="scene" data-composition-id="main" data-width="1920" data-height="1080"
41+
data-start="0" data-duration="4">
42+
<video id="clip" src="clip.mp4"></video>
43+
</div>
44+
<script src="gsap.min.js"></script>
45+
<script>
46+
window.__timelines = window.__timelines || {};
47+
window.__timelines["main"] = gsap.timeline({ paused: true });
48+
</script>
49+
</body></html>`;
50+
51+
function makeProject(html: string): string {
52+
const dir = mkdtempSync(join(tmpdir(), "hf-proj-"));
53+
mkdirSync(join(dir, "compositions"), { recursive: true });
54+
writeFileSync(join(dir, "index.html"), html, "utf-8");
55+
return dir;
56+
}
57+
58+
beforeEach(() => {
59+
enqueued.length = 0;
60+
rmSync(join(HOME, ".hyperframes"), { recursive: true, force: true });
61+
});
62+
63+
describe("trackLintRun end to end", () => {
64+
it("emits one lint_report carrying codes, timings, and the ruleset fingerprint", async () => {
65+
const dir = makeProject(COMPOSITION);
66+
const result = await lintProject(dir);
67+
trackLintRun(dir, result, { command: "lint", durationMs: 12 });
68+
69+
const reports = enqueued.filter((e) => e.event === "lint_report");
70+
expect(reports).toHaveLength(1);
71+
const props = reports[0]!.properties;
72+
73+
expect(props["command"]).toBe("lint");
74+
expect(props["files_scanned"]).toBe(1);
75+
expect(props["duration_ms"]).toBe(12);
76+
// The real linter found real problems in this composition.
77+
expect((props["codes"] as string[]).length).toBeGreaterThan(0);
78+
expect(props["error_count"]).toBeGreaterThan(0);
79+
// Timings are attributed per rule group and a slowest rule is identified.
80+
expect(Object.keys(props["rule_group_ms"] as object)).toContain("gsap");
81+
expect(props["slowest_rule"]).toMatch(/^[a-z]+#\d+$/);
82+
// Version and ruleset fingerprint ride along.
83+
expect(props["cli_version"]).toBeTruthy();
84+
expect(props["rule_count"]).toBeGreaterThan(0);
85+
// Per-group sizes make the positional `slowest_rule` index comparable
86+
// across builds: a group that changed size renumbered its rules.
87+
const groupCounts = props["rule_group_counts"] as Record<string, number>;
88+
expect(groupCounts["gsap"]).toBeGreaterThan(0);
89+
expect(Object.values(groupCounts).reduce((a, b) => a + b, 0)).toBe(props["rule_count"]);
90+
// code_counts sums to the number of findings.
91+
const counts = Object.values(props["code_counts"] as Record<string, number>);
92+
expect(counts.reduce((a, b) => a + b, 0)).toBe(
93+
result.results.flatMap((r) => r.result.findings).length,
94+
);
95+
96+
rmSync(dir, { recursive: true, force: true });
97+
});
98+
99+
it("emits lint_rule_streak with cleared:true once an edit removes the finding", async () => {
100+
const dir = makeProject(COMPOSITION);
101+
102+
const first = await lintProject(dir);
103+
trackLintRun(dir, first, { command: "lint", durationMs: 1 });
104+
const codes = first.results[0]!.result.findings.map((f) => f.code);
105+
expect(codes).toContain("media_missing_data_start");
106+
107+
// Fix exactly that finding and re-lint.
108+
writeFileSync(
109+
join(dir, "index.html"),
110+
COMPOSITION.replace('<video id="clip"', '<video id="clip" data-start="0" data-duration="4"'),
111+
"utf-8",
112+
);
113+
enqueued.length = 0;
114+
const second = await lintProject(dir);
115+
trackLintRun(dir, second, { command: "lint", durationMs: 1 });
116+
117+
const streaks = enqueued.filter((e) => e.event === "lint_rule_streak");
118+
const cleared = streaks.find((e) => e.properties["code"] === "media_missing_data_start");
119+
expect(cleared?.properties).toMatchObject({
120+
code: "media_missing_data_start",
121+
cleared: true,
122+
edits: 1,
123+
command: "lint",
124+
});
125+
126+
rmSync(dir, { recursive: true, force: true });
127+
});
128+
129+
it("never throws when the lint result is malformed", () => {
130+
expect(() =>
131+
trackLintRun("/nope", { results: [] } as never, { command: "lint", durationMs: 0 }),
132+
).not.toThrow();
133+
});
134+
});
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
// One place that turns a ProjectLintResult into telemetry, so `lint` and
2+
// `check` report identically instead of drifting apart.
3+
4+
import { LINT_RULE_COUNT, LINT_RULE_GROUP_COUNTS, type ProjectLintResult } from "@hyperframes/lint";
5+
import { trackLintReport, trackLintRuleStreak } from "./events.js";
6+
import { recordLintRun } from "./lintStreaks.js";
7+
8+
/**
9+
* Report one lint pass: aggregate counts and timings, plus any finding that
10+
* survived an edit to its file.
11+
*
12+
* Never throws — a telemetry failure must not fail the command that lints.
13+
*/
14+
export function trackLintRun(
15+
projectDir: string,
16+
lintResult: ProjectLintResult,
17+
options: { command: string; durationMs: number; runId?: string },
18+
): void {
19+
try {
20+
const runIdField = options.runId !== undefined ? { runId: options.runId } : {};
21+
22+
trackLintReport({
23+
command: options.command,
24+
durationMs: options.durationMs,
25+
filesScanned: lintResult.results.length,
26+
errorCount: lintResult.totalErrors,
27+
warningCount: lintResult.totalWarnings,
28+
infoCount: lintResult.totalInfos,
29+
ruleCount: LINT_RULE_COUNT,
30+
ruleGroupCounts: LINT_RULE_GROUP_COUNTS,
31+
...summarize(lintResult),
32+
...runIdField,
33+
});
34+
35+
const streaks = recordLintRun(
36+
projectDir,
37+
lintResult.results.map(({ file, contentHash, result }) => ({
38+
file,
39+
contentHash,
40+
findings: result.findings,
41+
})),
42+
);
43+
for (const streak of streaks) {
44+
trackLintRuleStreak({ ...streak, command: options.command, ...runIdField });
45+
}
46+
} catch {
47+
// Telemetry is best-effort. A malformed result, an unwritable home
48+
// directory, or a transport failure must never turn a green lint red.
49+
}
50+
}
51+
52+
/** Roll every file's findings and timings up into one run-level summary. */
53+
function summarize(lintResult: ProjectLintResult): {
54+
codeCounts: Record<string, number>;
55+
ruleGroupMs: Record<string, number>;
56+
slowestRule: string;
57+
slowestRuleMs: number;
58+
} {
59+
const codeCounts: Record<string, number> = {};
60+
const ruleGroupMs: Record<string, number> = {};
61+
let slowestRule = "";
62+
let slowestRuleMs = 0;
63+
64+
for (const { result } of lintResult.results) {
65+
for (const finding of result.findings) {
66+
codeCounts[finding.code] = (codeCounts[finding.code] ?? 0) + 1;
67+
}
68+
const timings = result.timings;
69+
if (!timings) continue;
70+
for (const [group, ms] of Object.entries(timings.groupMs)) {
71+
ruleGroupMs[group] = (ruleGroupMs[group] ?? 0) + ms;
72+
}
73+
if (timings.slowestRuleMs > slowestRuleMs) {
74+
slowestRuleMs = timings.slowestRuleMs;
75+
slowestRule = timings.slowestRule;
76+
}
77+
}
78+
79+
for (const group of Object.keys(ruleGroupMs)) {
80+
ruleGroupMs[group] = Math.round(ruleGroupMs[group]!);
81+
}
82+
return { codeCounts, ruleGroupMs, slowestRule, slowestRuleMs };
83+
}

0 commit comments

Comments
 (0)