Skip to content

Commit 6b5b4cb

Browse files
feat(studio): make agent edits live and explicit (#3581)
1 parent f498be9 commit 6b5b4cb

94 files changed

Lines changed: 7174 additions & 978 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/guides/webmcp.mdx‎

Lines changed: 51 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,16 @@ Studio registers its own capabilities as WebMCP tools, so an AI agent running in
1414

1515
## What it looks like
1616

17-
With the tools available, an agent can do this without touching your files:
17+
With the tools available, an agent can inspect the source-backed scene, make one targeted edit, and
18+
check the result:
1819

1920
```text
20-
studio_look -> the project, playhead, selection, and every element
21-
studio_select hf:abc123 -> selects the headline, same as clicking it
22-
studio_inspect -> its resolved styles, text, and animations
23-
studio_set_style {"color":"red"} -> writes it, through Studio's own commit path
24-
studio_frame 2.4 -> a PNG of the composition at 2.4 seconds
21+
studio_look -> find the headline and copy its handle
22+
studio_select {"handle":"hf:abc123"} -> share the target with the person in Studio
23+
studio_inspect {"handle":"hf:abc123"} -> read its styles, text, and capabilities
24+
studio_set_style {"handle":"hf:abc123",
25+
"styles":{"color":"red"}} -> write that source-backed element
26+
studio_frame {"time":2.4} -> capture the composition at 2.4 seconds
2527
```
2628

2729
The last one matters most. It is what lets an agent judge a change instead of guessing at it.
@@ -63,20 +65,24 @@ console.log(tools.map((tool) => tool.name));
6365
feature-detecting it will mislead you.
6466
</Warning>
6567

66-
On browsers without native support, Studio loads a polyfill so a WebMCP bridge extension can still
67-
connect. Nothing is downloaded on a browser that has the API already.
68+
On browsers without native support, Studio loads a bundled polyfill so a WebMCP bridge extension can
69+
still connect. Nothing is downloaded on a browser that has the API already. When the Studio
70+
preference is disabled, it registers no tools.
6871

6972
## What an agent can do
7073

7174
### Read
7275

7376
| Tool | Answers |
7477
| --- | --- |
75-
| `studio_look` | The open project and composition, the playhead, what you have selected, and every element with a handle |
78+
| `studio_look` | The open project and composition, the playhead, selection, undo state, and a bounded source-backed scene in nested order |
7679
| `studio_inspect` | One element in full: resolved styles, text fields, box, animations, and what it will accept |
7780
| `studio_frame` | A PNG of the composition at any time |
7881

79-
`studio_look` gives every element a **handle**. Pass it back to any tool that edits an element.
82+
Each element returned by `studio_look` has a **handle**. It also reports `sourceFile`, `parentHandle`,
83+
`depth`, and `childCount`, so an agent can distinguish the same authored ID in two nested scene
84+
files. Pass a handle back unchanged to every element write. Handles are source-safe addresses, not
85+
CSS selectors.
8086

8187
### Change
8288

@@ -92,29 +98,49 @@ connect. Nothing is downloaded on a browser that has the API already.
9298
| `studio_add_keyframe` | Adds a keyframe to an animation |
9399
| `studio_delete_animation` | Removes an animation |
94100

95-
Every edit runs through the same commit path a mouse gesture uses, so it lands in your file with the
96-
same undo entry and the same save behaviour. There is no separate agent write path.
101+
Element writes run through the same commit actors Studio uses. When that actor forwards versioned
102+
durability evidence, the receipt names the source file and content version, and the edit enters the
103+
same undo history. Actors without that evidence stay at `dispatched`.
97104

98105
## Two rules worth knowing
99106

100-
**Select first, then edit.** Most editing tools act on the current selection rather than taking an
101-
element. That is how Studio itself works: click, then type. An agent that edits without selecting
102-
gets an error telling it to select.
107+
**Show intent, then address every write explicitly.** Before the first write to a target, call
108+
`studio_select` so the person in Studio sees the same selection box and inspector as the agent.
109+
Selection communicates intent; it does not grant write authority. `studio_set_text`,
110+
`studio_set_style`, `studio_transform`, and the animation tools still require that target's handle
111+
from `studio_look`, so a later human click cannot redirect an already-addressed write.
103112

104-
**Check what came back.** Tools report what actually happened, not what was asked for.
105-
`studio_transform` reads the element's box back after writing and tells you which operations took
106-
effect. `studio_frame` reports the time it actually captured. When something could not be verified,
107-
the tool says so rather than claiming success.
113+
**Read the receipt stage.** A write result separates acceptance from proof:
114+
115+
| Stage | What it proves |
116+
| --- | --- |
117+
| `refused` | No commit actor ran. Fix the handle, input, capability, or Studio state before retrying. |
118+
| `dispatched` | The actor accepted the request, but the tool has no durable version or independent readback. Follow with `studio_inspect` or `studio_frame`. |
119+
| `saved` | Studio received versioned evidence that the named source file persisted the write. |
120+
| `verified` | The write was saved and an independent readback observed the result. |
121+
| `failed` | A commit actor ran and then failed. Inspect `kind`, `reason`, and any `hint`. |
122+
123+
`changed` is separate from the stage. A saved no-op is still truthful: the file accepted the request,
124+
but the value was already present. Style writes include a receipt per property and can be partial.
125+
Animation handlers currently report `dispatched` after their persistence and live-preview sync
126+
settle; they do not claim versioned durability or independent readback when the underlying handler
127+
cannot provide that evidence. A late cancellation request also does not undo an edit that was
128+
already dispatched or saved.
108129

109130
## Working alongside an agent
110131

111-
This is built for you and an agent looking at the same composition. Studio shows you every change as
112-
it happens: an agent selecting an element draws the same selection box, and an edit appears in your
113-
undo history under its own name.
132+
This is built for you and an agent looking at the same composition. Studio shows selection normally
133+
and adds a **Topology Lens** around the addressed element while a transaction is resolving. A new
134+
target is acquired, repeated edits localize more quickly, and a durable result seals. The lens is
135+
Studio-only transaction feedback: it is not added to authored HTML, the preview iframe, captured
136+
frames, or thumbnails.
137+
138+
When a write tool finishes successfully, the open Studio preview has already synchronized that
139+
edit. The person watching does not need to scrub the timeline or refresh the browser to see it.
114140

115-
That shared view is doing real work. Some of Studio's write paths report a failure through a toast
116-
rather than a return value, so **you** are the one who sees it. Leave Studio visible while an agent
117-
is working.
141+
After an accepted source-file change, Studio advances the project thumbnail revision and regenerates
142+
visible composition thumbnails. This lets the sidebar converge on the saved project instead of
143+
continuing to show cached pre-edit pixels.
118144

119145
<Note>
120146
Studio refuses agent writes while auto-save is paused or an external change to the file is waiting

‎packages/studio-server/src/helpers/projectSignature.test.ts‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,23 @@
1-
import { describe, expect, it } from "vitest";
1+
import { afterEach, describe, expect, it } from "vitest";
2+
import {
3+
closeSync,
4+
fstatSync,
5+
ftruncateSync,
6+
futimesSync,
7+
mkdtempSync,
8+
openSync,
9+
rmSync,
10+
writeSync,
11+
} from "node:fs";
12+
import { tmpdir } from "node:os";
213
import { resolve } from "node:path";
3-
import { affectsProjectSignature } from "./projectSignature.js";
14+
import { affectsProjectSignature, createProjectSignature } from "./projectSignature.js";
15+
16+
const temporaryProjects: string[] = [];
17+
18+
afterEach(() => {
19+
for (const project of temporaryProjects.splice(0)) rmSync(project, { recursive: true });
20+
});
421

522
const PROJECT = resolve("/projects/demo");
623
const affects = (relativePath: string) =>
@@ -42,3 +59,25 @@ describe("affectsProjectSignature", () => {
4259
expect(affectsProjectSignature(PROJECT, PROJECT)).toBe(false);
4360
});
4461
});
62+
63+
describe("createProjectSignature", () => {
64+
it("changes after same-size content is written with the original mtime restored", () => {
65+
const project = mkdtempSync(resolve(tmpdir(), "hf-signature-"));
66+
temporaryProjects.push(project);
67+
const file = resolve(project, "index.html");
68+
const descriptor = openSync(file, "w+");
69+
try {
70+
writeSync(descriptor, "first");
71+
const originalMtime = fstatSync(descriptor).mtime;
72+
const before = createProjectSignature(project);
73+
74+
ftruncateSync(descriptor, 0);
75+
writeSync(descriptor, "other", 0, "utf8");
76+
futimesSync(descriptor, originalMtime, originalMtime);
77+
78+
expect(createProjectSignature(project)).not.toBe(before);
79+
} finally {
80+
closeSync(descriptor);
81+
}
82+
});
83+
});

‎packages/studio-server/src/helpers/projectSignature.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ export function affectsProjectSignature(projectDir: string, changedPath: string)
7070
interface ProjectSignatureFile {
7171
file: string;
7272
mtimeMs: number;
73+
ctimeMs: number;
7374
size: number;
7475
textContentEligible: boolean;
7576
}
@@ -124,6 +125,7 @@ function collectProjectSignatureFiles(
124125
files.push({
125126
file,
126127
mtimeMs: stat.mtimeMs,
128+
ctimeMs: stat.ctimeMs,
127129
size: stat.size,
128130
textContentEligible: isTextContentEligible(file, stat.size),
129131
});
@@ -149,6 +151,7 @@ function collectProjectSignatureManifestFiles(
149151
files.push({
150152
file,
151153
mtimeMs: stat.mtimeMs,
154+
ctimeMs: stat.ctimeMs,
152155
size: stat.size,
153156
textContentEligible: isTextContentEligible(file, stat.size),
154157
});
@@ -165,6 +168,8 @@ function createProjectFingerprint(projectDir: string, files: ProjectSignatureFil
165168
hash.update("\0");
166169
hash.update(String(entry.mtimeMs));
167170
hash.update("\0");
171+
hash.update(String(entry.ctimeMs));
172+
hash.update("\0");
168173
hash.update(entry.textContentEligible ? "text" : "binary");
169174
hash.update("\0");
170175
}

‎packages/studio-server/src/routes/files.test.ts‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -437,18 +437,59 @@ describe("registerFileRoutes", () => {
437437
const payload = (await response.json()) as {
438438
changed?: boolean;
439439
path?: string;
440+
version?: string;
440441
backupPath?: string;
441442
};
442443

443444
expect(payload.changed).toBe(true);
444445
expect(payload.path).toBe("index.html");
446+
expect(payload.version).toBe(
447+
fileContentVersion(readFileSync(join(projectDir, "index.html"), "utf-8")),
448+
);
445449
expect(payload.backupPath).toMatch(/^\.hyperframes\/backup\//);
446450
expect(readFileSync(join(projectDir, payload.backupPath!), "utf-8")).toBe(
447451
'<div id="title">Before</div>',
448452
);
449453
expect(readFileSync(join(projectDir, "index.html"), "utf-8")).toContain("After");
450454
});
451455

456+
it("returns the current durable version for a matched no-op element patch", async () => {
457+
const projectDir = createProjectDir();
458+
const original = '<div id="title">Before</div>';
459+
writeFileSync(join(projectDir, "index.html"), original);
460+
const app = new Hono();
461+
registerFileRoutes(app, createAdapter(projectDir));
462+
463+
const response = await app.request(
464+
"http://localhost/projects/demo/file-mutations/patch-element/index.html",
465+
{
466+
method: "POST",
467+
headers: { "Content-Type": "application/json" },
468+
body: JSON.stringify({
469+
target: { id: "title" },
470+
operations: [{ type: "text-content", property: "textContent", value: "Before" }],
471+
}),
472+
},
473+
);
474+
const payload = (await response.json()) as {
475+
changed?: boolean;
476+
matched?: boolean;
477+
path?: string;
478+
version?: string;
479+
backupPath?: string;
480+
};
481+
482+
expect(response.status).toBe(200);
483+
expect(payload).toMatchObject({
484+
changed: false,
485+
matched: true,
486+
path: "index.html",
487+
version: fileContentVersion(original),
488+
});
489+
expect(payload.backupPath).toBeUndefined();
490+
expect(existsSync(join(projectDir, ".hyperframes", "backup"))).toBe(false);
491+
});
492+
452493
// Without the receipt the client cannot recognise its own edit in the watcher
453494
// broadcast, so it treats it as someone else's write and does a full preview
454495
// reload — a visible blank on the stage right after the user typed.
@@ -1292,6 +1333,30 @@ const tl = gsap.timeline({ paused: true });
12921333
expect(res.status).toBe(400);
12931334
});
12941335

1336+
it("rejects raw JavaScript expressions at the GSAP mutation boundary", async () => {
1337+
const projectDir = createProjectDir();
1338+
writeHtml(projectDir, "comp.html", FROMTO_COMP);
1339+
const app = new Hono();
1340+
registerFileRoutes(app, createAdapter(projectDir));
1341+
const animation = await getFirstAnimation(app, "comp.html");
1342+
1343+
const response = await app.request("http://localhost/projects/demo/gsap-mutations/comp.html", {
1344+
method: "POST",
1345+
headers: { "Content-Type": "application/json" },
1346+
body: JSON.stringify({
1347+
type: "update-meta",
1348+
animationId: animation.id,
1349+
updates: { ease: "__raw:(()=>alert(1))()" },
1350+
}),
1351+
});
1352+
1353+
expect(response.status).toBe(400);
1354+
expect(await response.json()).toEqual({
1355+
error: "raw JavaScript expressions are not accepted",
1356+
});
1357+
expect(readFileSync(join(projectDir, "comp.html"), "utf8")).toBe(FROMTO_COMP);
1358+
});
1359+
12951360
it("update-from-property updates a fromTo start value in place", async () => {
12961361
const projectDir = createProjectDir();
12971362
writeHtml(projectDir, "comp.html", FROMTO_COMP);

‎packages/studio-server/src/routes/files.ts‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1186,6 +1186,9 @@ function validateGsapMutationRequest(
11861186
if (!body || typeof body !== "object" || !("type" in body) || !body.type) {
11871187
return c.json({ error: "mutation type required" }, 400);
11881188
}
1189+
if (containsRawGsapExpression(body)) {
1190+
return c.json({ error: "raw JavaScript expressions are not accepted" }, 400);
1191+
}
11891192
const unsafeFields = findUnsafeMutationValues(body);
11901193
if (unsafeFields.length > 0) return rejectUnsafeMutationValues(c, unsafeFields);
11911194
if (
@@ -1197,6 +1200,13 @@ function validateGsapMutationRequest(
11971200
return null;
11981201
}
11991202

1203+
function containsRawGsapExpression(value: unknown): boolean {
1204+
if (typeof value === "string") return value.startsWith("__raw:");
1205+
if (Array.isArray(value)) return value.some(containsRawGsapExpression);
1206+
if (!value || typeof value !== "object") return false;
1207+
return Object.values(value).some(containsRawGsapExpression);
1208+
}
1209+
12001210
async function prepareGsapMutationScript(
12011211
c: RouteContext,
12021212
res: ResolvedGsapFile,
@@ -2769,27 +2779,32 @@ export function registerFileRoutes(api: Hono, adapter: StudioApiAdapter): void {
27692779
parsed.body.operations,
27702780
);
27712781
if (patched === originalContent) {
2782+
const version = fileContentVersion(originalContent);
2783+
c.header("ETag", version);
27722784
return c.json({
27732785
ok: true,
27742786
changed: false,
27752787
matched,
27762788
content: originalContent,
27772789
path: ctx.filePath,
2790+
version,
27782791
});
27792792
}
2780-
const { backupPath } = writeMutationResult(
2793+
const { backupPath, version } = writeMutationResult(
27812794
c,
27822795
ctx.project.dir,
27832796
ctx.filePath,
27842797
ctx.absPath,
27852798
patched,
27862799
);
2800+
c.header("ETag", version);
27872801
return c.json({
27882802
ok: true,
27892803
changed: true,
27902804
matched,
27912805
content: patched,
27922806
path: ctx.filePath,
2807+
version,
27932808
backupPath,
27942809
});
27952810
});

0 commit comments

Comments
 (0)