Skip to content

Commit 23d1fe5

Browse files
authored
feat(cli): save check, snapshot and render artifacts per revision (#5522)
* feat(cli): save check, snapshot and render artifacts per revision * fix(cli): keep clean --snapshots inside the project A symlinked .hyperframes or artifacts folder was followed, so the command deleted snapshot and check files outside the project. * test(cli): keep check, snapshot, and render successful when recording fails A held history or a corrupt output registry warns and leaves the command result alone. * fix(cli): drop snapshot records when a replaced run is not recorded A reused --output rewrites the files even when history is busy. Those records are removed so the old revision does not keep the new pixels. The preview checkpoint stops waiting after five seconds. A bad PNG header no longer fails the snapshot. * fix(studio-server): keep the video-state comment share from rising The forget helper's doc comment pushed the file above its base share. * fix(cli): post the preview checkpoint and keep cleanup inside artifacts A preview checkpoint with no body was sent as GET, so Studio never ran it. Cleanup now requires the real path to stay under the artifacts folder. A failed follow-up drop does not warn twice. The studio-server unpacked budget covers the record-dropping helper. * docs(cli): shorten the snapshot skill notes The command examples already show range and composition. The prose only keeps the output path and the flags that are easy to misuse. * chore: leave package headroom for artifact cleanup * test(cli): compare artifact paths portably * style(cli): format portable artifact assertion
1 parent fb4189c commit 23d1fe5

31 files changed

Lines changed: 793 additions & 61 deletions

‎docs/developers/harness-tool-context.mdx‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,19 +46,20 @@ Each line is an event:
4646
"version": 1,
4747
"type": "artifact.created",
4848
"command": "snapshot",
49-
"artifact": { "role": "contact-sheet", "path": "/project/snapshots/contact-sheet.jpg", "mediaType": "image/jpeg" }
49+
"artifact": { "role": "contact-sheet", "path": "/project/.hyperframes/artifacts/snapshots/…/contact-sheet.jpg", "mediaType": "image/jpeg" }
5050
}
5151
```
5252

5353
| Command | `role` | What it is |
5454
| --- | --- | --- |
5555
| `snapshot` | `frame` | Each captured frame (PNG) |
5656
| `snapshot` | `contact-sheet` | Each page of the grid of frames made for review (JPEG) |
57+
| `check` | `check-report` | The saved JSON report |
5758
| `check` | `frame` | Each frame the browser check captured |
5859
| `check` | `finding` | Each crop around a finding |
5960

6061
Paths are absolute and stay where the command saved them. What to do with an artifact is the
61-
harness's decision: attach the contact sheet to the tool result, or ignore it.
62+
harness's decision: attach the contact sheet to the tool result, link the report, or ignore it.
6263
Ignore event types and roles you do not know; new ones are added without a version change.
6364

6465
## `project-lock`: one browser command at a time

‎docs/packages/cli.mdx‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -737,7 +737,7 @@ cover every finding from the full check.
737737
| Flag | Description |
738738
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
739739
| `--json` | One aggregated envelope. Every finding carries a selector, `data-*` identity, source file, bbox, and sample time. |
740-
| `--snapshots` | Save the five audited frames under `snapshots/`, plus a `finding-NN-<code>.png` crop per finding |
740+
| `--snapshots` | Save the five audited frames under `.hyperframes/artifacts/snapshots/`, plus a `finding-NN-<code>.png` crop per finding |
741741
| `--samples` | Midpoint samples across the duration (default 9) |
742742
| `--at` | Explicit timestamps in seconds, comma-separated |
743743
| `--at-transitions` | Also sample every tween start and end, to catch transient overlaps at transition seams |
@@ -854,10 +854,10 @@ npx hyperframes snapshot my-project --at 1.5,4.3 --against ref.mp4
854854
```
855855
◆ Capturing 3 frames at [2.9s, 10.4s, 18.7s] from my-project
856856
857-
◇ 3 snapshots saved to snapshots/
858-
snapshots/frame-00-at-2.9s.png
859-
snapshots/frame-01-at-10.4s.png
860-
snapshots/frame-02-at-18.7s.png
857+
◇ 3 snapshots saved to <project>/.hyperframes/artifacts/snapshots/<id>
858+
.hyperframes/artifacts/snapshots/<id>/frame-00-at-2.9s.png
859+
.hyperframes/artifacts/snapshots/<id>/frame-01-at-10.4s.png
860+
.hyperframes/artifacts/snapshots/<id>/frame-02-at-18.7s.png
861861
```
862862

863863
| Flag | Description |
@@ -866,7 +866,7 @@ npx hyperframes snapshot my-project --at 1.5,4.3 --against ref.mp4
866866
| `--frames` | Evenly spaced frames instead (default 5) |
867867
| `--range <start,end>` | Spread `--frames` across this interval only, in root-timeline seconds, with no end-of-timeline frame added |
868868
| `--composition <id>` | Spread `--frames` across the visible interval of the element with this `data-composition-id`. Use one of `--at`, `--range` or `--composition`. |
869-
| `--output, -o` | Where the PNGs go (default `<project>/snapshots`) |
869+
| `--output, -o` | Where the PNGs go (default: a new `<project>/.hyperframes/artifacts/snapshots/<id>` per run) |
870870
| `--end` / `--no-end` | Always add a readable end-of-timeline frame (default: on). `--no-end` captures only your `--at` times. |
871871
| `--zoom` | Crop to a CSS selector or an exact `x,y,w,h` region, using a raised device scale factor. Layout is untouched — never CSS zoom or a resized viewport. A selector that matches nothing is an error. |
872872
| `--against` | A reference video, such as footage you are rebuilding. For every captured time also saves its frame-exact frame (`ref-XX-…png`) and a labelled render / reference pair (`pair-XX-…jpg`). |
@@ -1345,7 +1345,7 @@ Find and remove what HyperFrames left on disk.
13451345
```bash
13461346
npx hyperframes clean --dry-run # list it, remove nothing
13471347
npx hyperframes clean
1348-
npx hyperframes clean --snapshots # also remove snapshots/ folders
1348+
npx hyperframes clean --snapshots # also remove snapshots and saved check reports
13491349
npx hyperframes clean --json
13501350
```
13511351

@@ -1356,7 +1356,7 @@ It looks in the current project (not its subfolders) and every project the Studi
13561356
- the history of projects that no longer exist;
13571357
- extracted video frames and video proxies nobody has used for an hour (they rebuild on demand).
13581358

1359-
`snapshots/` folders are listed as also reclaimable and removed only with `--snapshots`. Rendered outputs, source
1359+
`snapshots/` folders and the saved snapshots and check reports under `.hyperframes/artifacts/` are listed as also reclaimable and removed only with `--snapshots`. Rendered outputs, source
13601360
files, the browser download and models are never touched.
13611361

13621362
### `info` and `compositions`

‎packages/cli/src/commands/check.ts‎

Lines changed: 105 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,14 @@
1-
import { resolve } from "node:path";
1+
import { captureArtifactRevision } from "../utils/artifactCapture.js";
2+
import { pngDimensions } from "../utils/pngDimensions.js";
3+
import { randomUUID } from "node:crypto";
4+
import { mkdirSync, writeFileSync } from "node:fs";
5+
import { join, resolve } from "node:path";
6+
import {
7+
recordArtifacts,
8+
type ArtifactCapture,
9+
type ArtifactInput,
10+
type ArtifactRecord,
11+
} from "@hyperframes/studio-server/history";
212
import { formatFindingTimes } from "../utils/checkFindings.js";
313
import { withProjectLock } from "../utils/projectLock.js";
414
import { reportArtifacts } from "../utils/toolContext.js";
@@ -17,6 +27,7 @@ import {
1727
DEFAULT_CHECK_OPTIONS,
1828
checkExitCode,
1929
runCheckPipeline,
30+
selectFindingCropRequests,
2031
type CheckFinding,
2132
type CheckOptions,
2233
type CheckReport,
@@ -32,24 +43,108 @@ export const examples: Example[] = [
3243
["Also fail on warnings", "hyperframes check --strict"],
3344
];
3445

46+
type SavedCheck = Pick<ArtifactRecord, "id" | "revision" | "report" | "attribution">;
47+
type SavedCheckReport = CheckReport & { saved?: SavedCheck };
48+
3549
export interface CheckCommandDependencies {
3650
resolveProject(dir: string | undefined): ProjectDir;
37-
runPipeline(project: ProjectDir, options: CheckOptions): Promise<CheckReport>;
51+
runPipeline(project: ProjectDir, options: CheckOptions): Promise<SavedCheckReport>;
3852
withMeta(value: object): object;
3953
}
4054

4155
const DEFAULT_COMMAND_DEPENDENCIES: CheckCommandDependencies = {
4256
resolveProject,
4357
async runPipeline(project, options) {
58+
const capture = await captureArtifactRevision(project.dir).catch(warnNotRecorded);
4459
const report = await runCheckPipeline(project, options);
4560
const images = (files: readonly string[]) => files.map((file) => resolve(project.dir, file));
4661
reportArtifacts("check", "frame", images(report.snapshots.files));
4762
reportArtifacts("check", "finding", images(report.snapshots.findingFiles));
48-
return report;
63+
if (!capture) return report;
64+
try {
65+
return { ...report, saved: await saveCheckReport(project, options, report, capture) };
66+
} catch (error) {
67+
warnNotRecorded(error);
68+
return report;
69+
}
4970
},
5071
withMeta,
5172
};
5273

74+
function warnNotRecorded(error: unknown): undefined {
75+
console.warn(`${c.warn("⚠")} Check report was not recorded: ${normalizeErrorMessage(error)}`);
76+
return undefined;
77+
}
78+
79+
async function saveCheckReport(
80+
project: ProjectDir,
81+
options: CheckOptions,
82+
report: CheckReport,
83+
capture: ArtifactCapture,
84+
): Promise<SavedCheck> {
85+
const id = randomUUID();
86+
const directory = join(project.dir, ".hyperframes", "artifacts", "checks");
87+
mkdirSync(directory, { recursive: true });
88+
const path = join(directory, `${id}.json`);
89+
writeFileSync(path, JSON.stringify(report, null, 2) + "\n", { flag: "wx" });
90+
const reportScope =
91+
"scope" in report && typeof report.scope === "object" && report.scope !== null
92+
? report.scope
93+
: undefined;
94+
const artifacts: ArtifactInput[] = [
95+
{
96+
kind: "check",
97+
id,
98+
scope: reportScope ? "scoped" : "project",
99+
report: resolve(path),
100+
metadata: {
101+
ok: report.ok,
102+
browserSkipped: report.browserSkipped,
103+
scope: reportScope,
104+
options,
105+
},
106+
},
107+
];
108+
for (const [index, image] of report.snapshots.files.entries()) {
109+
artifacts.push({
110+
kind: "snapshot",
111+
id: `${id}-${index}`,
112+
scope: "full",
113+
image: resolve(project.dir, image),
114+
time: report.snapshots.times[index],
115+
metadata: {
116+
checkId: id,
117+
...pngDimensions(resolve(project.dir, image)),
118+
},
119+
});
120+
}
121+
const crops = selectFindingCropRequests(report);
122+
for (const [index, image] of report.snapshots.findingFiles.entries()) {
123+
const crop = crops.find((request) => image.endsWith(request.filename));
124+
artifacts.push({
125+
kind: "snapshot",
126+
id: `${id}-crop-${index}`,
127+
scope: "finding",
128+
image: resolve(project.dir, image),
129+
time: crop?.time,
130+
metadata: {
131+
checkId: id,
132+
finding: crop,
133+
...pngDimensions(resolve(project.dir, image)),
134+
},
135+
});
136+
}
137+
const [saved] = await recordArtifacts(project.dir, capture, artifacts);
138+
if (!saved) throw new Error("no history record was written");
139+
reportArtifacts("check", "check-report", [resolve(path)]);
140+
return {
141+
id: saved.id,
142+
revision: saved.revision,
143+
report: saved.report,
144+
attribution: saved.attribution,
145+
};
146+
}
147+
53148
const CHECK_COMMAND_ARGS = {
54149
dir: { type: "positional", description: "Project directory", required: false },
55150
json: { type: "boolean", description: "Output agent-readable JSON", default: false },
@@ -128,7 +223,7 @@ const CHECK_COMMAND_ARGS = {
128223
},
129224
snapshots: {
130225
type: "boolean",
131-
description: "Save the five contrast-pass PNGs under snapshots/",
226+
description: "Save contrast-pass PNGs under .hyperframes/artifacts/snapshots/",
132227
default: false,
133228
},
134229
"caption-zone": {
@@ -174,6 +269,12 @@ export function createCheckCommand(
174269
console.log(JSON.stringify(dependencies.withMeta(report), null, 2));
175270
} else {
176271
printHumanReport(report);
272+
if (report.saved) {
273+
console.log(`Report: ${report.saved.report}`);
274+
console.log(
275+
`Revision: ${report.saved.revision ?? "none"} (${report.saved.attribution})`,
276+
);
277+
}
177278
}
178279
setCommandExitCode(checkExitCode(report));
179280
} catch (error) {
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
2+
import { openProjectHistory } from "@hyperframes/studio-server/history";
3+
import { historyDeps } from "../utils/historyOwner.js";
4+
import { tmpdir } from "node:os";
5+
import { join } from "node:path";
6+
import { runCommand } from "citty";
7+
import { expect, it, vi } from "vitest";
8+
import check from "./check.js";
9+
import { consumeCommandResult } from "../utils/commandResult.js";
10+
import { checkExitCode, type CheckReport } from "../utils/checkPipeline.js";
11+
12+
it("persists real lint-blocked reports without overwriting earlier checks or inventing a revision", async () => {
13+
const project = mkdtempSync(join(tmpdir(), "hf-check-artifacts-"));
14+
const log = vi.spyOn(console, "log").mockImplementation(() => undefined);
15+
writeFileSync(
16+
join(project, "index.html"),
17+
"<html><body>Missing composition metadata</body></html>",
18+
);
19+
try {
20+
for (let i = 0; i < 2; i++) {
21+
await runCommand(check, { rawArgs: [project, "--json"] });
22+
consumeCommandResult();
23+
}
24+
const state = JSON.parse(readFileSync(join(project, ".hyperframes/video-state.json"), "utf8"));
25+
expect(state.checks.unversioned).toHaveLength(2);
26+
const [first, second] = state.checks.unversioned;
27+
expect(first.report).not.toBe(second.report);
28+
expect(first.report).toBe(
29+
join(project, ".hyperframes", "artifacts", "checks", `${first.id}.json`),
30+
);
31+
expect(JSON.parse(readFileSync(first.report, "utf8")).browserSkipped).toBe(true);
32+
expect(state.checks.latest).toEqual({ revision: null, id: second.id });
33+
expect(state.checks.revisions).toEqual({});
34+
const output = JSON.parse(String(log.mock.calls.at(-1)?.[0]));
35+
expect(output.saved).toEqual({
36+
id: second.id,
37+
revision: null,
38+
report: second.report,
39+
attribution: "unversioned",
40+
});
41+
expect(output.browserSkipped).toBe(true);
42+
} finally {
43+
log.mockRestore();
44+
consumeCommandResult();
45+
rmSync(project, { recursive: true, force: true });
46+
}
47+
});
48+
49+
it("only warns when the report cannot be recorded; the exit status is the check's own", async () => {
50+
const project = mkdtempSync(join(tmpdir(), "hf-check-unrecorded-"));
51+
const log = vi.spyOn(console, "log").mockImplementation(() => undefined);
52+
const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);
53+
writeFileSync(
54+
join(project, "index.html"),
55+
"<html><body>Missing composition metadata</body></html>",
56+
);
57+
mkdirSync(join(project, ".hyperframes", "artifacts"), { recursive: true });
58+
writeFileSync(join(project, ".hyperframes", "artifacts", "checks"), "not a directory");
59+
try {
60+
await runCommand(check, { rawArgs: [project, "--json"] });
61+
const output = JSON.parse(String(log.mock.calls.at(-1)?.[0])) as CheckReport;
62+
expect(output).not.toHaveProperty("saved");
63+
expect(output.browserSkipped).toBe(true);
64+
expect(consumeCommandResult().exitCode).toBe(checkExitCode(output));
65+
expect(String(warn.mock.calls.at(-1)?.[0])).toContain("Check report was not recorded");
66+
} finally {
67+
log.mockRestore();
68+
warn.mockRestore();
69+
consumeCommandResult();
70+
rmSync(project, { recursive: true, force: true });
71+
}
72+
});
73+
74+
it("still reports the check when another process holds history", async () => {
75+
const project = mkdtempSync(join(tmpdir(), "hf-check-history-held-"));
76+
const log = vi.spyOn(console, "log").mockImplementation(() => undefined);
77+
const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);
78+
writeFileSync(
79+
join(project, "index.html"),
80+
"<html><body>Missing composition metadata</body></html>",
81+
);
82+
const previousWait = historyDeps.ownerWaitMs;
83+
historyDeps.ownerWaitMs = 0;
84+
const held = await openProjectHistory({
85+
projectDir: project,
86+
historyRoot: historyDeps.historyRoot,
87+
ownerWaitMs: 0,
88+
});
89+
try {
90+
await runCommand(check, { rawArgs: [project, "--json"] });
91+
const output = JSON.parse(String(log.mock.calls.at(-1)?.[0])) as CheckReport;
92+
expect(output).not.toHaveProperty("saved");
93+
expect(output.browserSkipped).toBe(true);
94+
expect(consumeCommandResult().exitCode).toBe(checkExitCode(output));
95+
expect(
96+
warn.mock.calls.some((call) => String(call[0]).includes("Check report was not recorded")),
97+
).toBe(true);
98+
} finally {
99+
historyDeps.ownerWaitMs = previousWait;
100+
await held.close();
101+
log.mockRestore();
102+
warn.mockRestore();
103+
consumeCommandResult();
104+
rmSync(project, { recursive: true, force: true });
105+
}
106+
});

0 commit comments

Comments
 (0)