Skip to content

Commit 3ccf2a0

Browse files
authored
feat: render inline web chat widgets via capability-gated show_widget tool (#101840)
Adds client-capability-gated tool availability: gateway clients declare capabilities at connect (new inline-widgets cap), chat.send stamps them into the run context, and every tool assembly path (embedded runner, queued followups, Codex app-server harness, plugin-only construction plans) drops tools whose requiredClientCaps the originating client did not declare. The Canvas plugin ships the first such tool, show_widget: agents pass SVG or an HTML fragment plus a title; the plugin hosts it as a bounded, retention-scoped Canvas document and returns the existing canvas preview handle, which web chat renders as a sandboxed iframe fitted to the widget's reported content height. Widget frames never get allow-same-origin (per-preview sandbox ceiling, including the sidebar path) and the Canvas host serves widget documents with a CSP sandbox header so direct navigation runs in an opaque origin. Verified live end-to-end on a Testbox with gpt-5.5 (screenshots on the PR). CLI-backed model backends do not carry client caps yet and stay fail-closed (#102577). Closes #101790
1 parent de2af01 commit 3ccf2a0

63 files changed

Lines changed: 1038 additions & 32 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/docs.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1330,6 +1330,7 @@
13301330
"tools/btw",
13311331
"tools/code-execution",
13321332
"tools/diffs",
1333+
"tools/show-widget",
13331334
"tools/elevated",
13341335
"tools/permission-modes",
13351336
"tools/exec-approvals",

docs/docs_map.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3606,6 +3606,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
36063606
- Headings:
36073607
- H2: Transport and framing
36083608
- H2: Handshake
3609+
- H3: Client capabilities
36093610
- H3: Node connect example
36103611
- H2: Roles and scopes
36113612
- H2: Presence
@@ -9623,6 +9624,14 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
96239624
- H2: Notes
96249625
- H2: Related
96259626

9627+
## tools/show-widget.md
9628+
9629+
- Route: /tools/show-widget
9630+
- Headings:
9631+
- H2: Use the tool
9632+
- H2: Security and storage
9633+
- H2: Related
9634+
96269635
## tools/skill-workshop.md
96279636

96289637
- Route: /tools/skill-workshop

docs/gateway/protocol.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,15 @@ stale CLI/device pairing baselines blocking local backend work. Remote,
167167
browser-origin, node, and explicit device-token/device-identity clients still
168168
go through normal pairing and scope-upgrade checks.
169169

170+
### Client capabilities
171+
172+
Operator clients may advertise optional capabilities in `connect.params.caps`:
173+
174+
- `tool-events`: accepts structured tool lifecycle events.
175+
- `inline-widgets`: can render hosted inline widget tool results.
176+
177+
Client capabilities describe the connected client, not authorization. Agent tools may declare required capabilities; the Gateway omits those tools unless every requirement appears in the originating client's `caps`. Channel-originated runs have no Gateway client capabilities, so capability-gated tools are unavailable even when tool policy explicitly allows them.
178+
170179
### Node connect example
171180

172181
```json

docs/tools/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ for contract details.
109109
Common plugin-provided tools include:
110110

111111
- [Diffs](/tools/diffs) for rendering file and markdown diffs
112+
- [Show widget](/tools/show-widget) for self-contained inline SVG and HTML in web chat
112113
- [LLM Task](/tools/llm-task) for JSON-only workflow steps
113114
- [Lobster](/tools/lobster) for typed workflows with resumable approvals
114115
- [Tokenjuice](/tools/tokenjuice) for compacting noisy `exec` and `bash` tool

docs/tools/show-widget.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
summary: "Render self-contained SVG or HTML widgets inline in web chat"
3+
title: "Show widget"
4+
sidebarTitle: "Show widget"
5+
read_when:
6+
- You want an agent to render an interactive result inside web chat
7+
- You need the show_widget input, security, or retention contract
8+
---
9+
10+
`show_widget` renders a self-contained SVG or HTML fragment inline in the Control UI chat transcript. The bundled Canvas plugin owns the tool and hosts each result as a same-origin Canvas document.
11+
12+
The tool is available only when the originating Gateway client declares the `inline-widgets` capability. The Control UI declares this capability automatically. Channel runs such as Telegram and WhatsApp do not receive `show_widget`.
13+
14+
Capability transport currently covers the embedded runner and the Codex app-server backend. CLI-backed model backends do not yet carry client capabilities, so capability-gated tools stay unavailable (fail closed) on those backends.
15+
16+
## Use the tool
17+
18+
The agent supplies two required strings:
19+
20+
<ParamField path="title" type="string" required>
21+
Short title shown with the inline preview and in the hosted document title.
22+
</ParamField>
23+
24+
<ParamField path="widget_code" type="string" required>
25+
Self-contained SVG or HTML fragment. Input beginning with `<svg` after trimming is rendered in SVG mode; all other input is treated as an HTML fragment. Maximum length: 262,144 characters.
26+
</ParamField>
27+
28+
The tool result includes a Canvas preview handle, so web chat renders the widget directly from the tool call and restores it after history reload. Transcripts that do not render previews still show the hosted Canvas path.
29+
30+
## Security and storage
31+
32+
Widget documents use a restrictive Content Security Policy: inline style and script are allowed, images may use `data:` URLs, and external fetches and resource loads are blocked. Keep all markup, styles, scripts, and image data inside `widget_code`.
33+
34+
The iframe always omits `allow-same-origin`, even when the Control UI's global embed mode is `trusted`, so widget scripts cannot read the parent application origin. The Canvas host also serves widget documents with a `Content-Security-Policy: sandbox allow-scripts` response header, so opening the hosted URL directly still runs the widget in an opaque origin instead of the Control UI origin. Browser sandboxing does not prevent a script from navigating its own iframe; only render widget code you are willing to execute in that isolated frame.
35+
36+
The iframe also follows [`gateway.controlUi.embedSandbox`](/web/control-ui#hosted-embeds). The default `scripts` tier supports interactive widgets while preserving origin isolation.
37+
38+
Canvas retains at most 32 widgets per session (or per agent when no session is available). Creating another widget removes the oldest document in that scope.
39+
40+
## Related
41+
42+
- [Control UI hosted embeds](/web/control-ui#hosted-embeds)
43+
- [Canvas plugin](/plugins/reference/canvas)
44+
- [Gateway protocol client capabilities](/gateway/protocol#client-capabilities)

docs/web/control-ui.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -328,6 +328,8 @@ Web Push is independent of the iOS APNS relay path (see [Configuration](/gateway
328328

329329
Assistant messages can render hosted web content inline with the `[embed ...]` shortcode. The iframe sandbox policy is controlled by `gateway.controlUi.embedSandbox`:
330330

331+
The bundled Canvas plugin also provides [`show_widget`](/tools/show-widget) to render self-contained SVG or HTML directly from a tool call. The browser advertises the `inline-widgets` Gateway capability, and the resulting Canvas document remains available when chat history reloads. Channel-originated runs do not receive this tool.
332+
331333
<Tabs>
332334
<Tab title="strict">
333335
Disables script execution inside hosted embeds.

extensions/canvas/index.ts

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ import { createLazyRuntimeModule } from "openclaw/plugin-sdk/lazy-runtime";
99
import { definePluginEntry, type AnyAgentTool } from "openclaw/plugin-sdk/plugin-entry";
1010
import { canvasConfigSchema, isCanvasHostEnabled } from "./src/config.js";
1111
import { A2UI_PATH, CANVAS_HOST_PATH, CANVAS_WS_PATH } from "./src/host/a2ui-shared.js";
12-
import { CanvasToolSchema } from "./src/tool-schema.js";
12+
import {
13+
CanvasToolSchema,
14+
SHOW_WIDGET_REQUIRED_CLIENT_CAPS,
15+
ShowWidgetToolSchema,
16+
} from "./src/tool-schema.js";
1317

1418
const CANVAS_NODE_COMMANDS = [
1519
"canvas.present",
@@ -45,6 +49,26 @@ function createLazyCanvasTool(params: {
4549
};
4650
}
4751

52+
function createLazyShowWidgetTool(params: {
53+
config?: OpenClawConfig;
54+
sessionId?: string;
55+
agentId?: string;
56+
}): AnyAgentTool {
57+
const loadTool = createLazyRuntimeModule(() =>
58+
import("./src/widget-tool.js").then(({ createShowWidgetTool }) => createShowWidgetTool(params)),
59+
);
60+
return {
61+
label: "Show Widget",
62+
name: "show_widget",
63+
description:
64+
"Render self-contained SVG or HTML inline in web chat. Use for visual or interactive results; external resources are blocked, so inline all required code and data.",
65+
parameters: ShowWidgetToolSchema,
66+
requiredClientCaps: SHOW_WIDGET_REQUIRED_CLIENT_CAPS,
67+
execute: async (...args: Parameters<AnyAgentTool["execute"]>) =>
68+
await (await loadTool()).execute(...args),
69+
};
70+
}
71+
4872
export default definePluginEntry({
4973
id: "canvas",
5074
name: "Canvas",
@@ -127,6 +151,17 @@ export default definePluginEntry({
127151
workspaceDir: ctx.workspaceDir,
128152
}),
129153
);
154+
api.registerTool(
155+
(ctx) =>
156+
isCanvasHostEnabled(ctx.runtimeConfig ?? ctx.config)
157+
? createLazyShowWidgetTool({
158+
config: ctx.runtimeConfig ?? ctx.config,
159+
sessionId: ctx.sessionId,
160+
agentId: ctx.agentId,
161+
})
162+
: null,
163+
{ name: "show_widget" },
164+
);
130165
api.registerNodeCliFeature(
131166
async ({ program }) => {
132167
const { createDefaultCanvasCliDependencies, registerNodesCanvasCommands } =

extensions/canvas/openclaw.plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"description": "Experimental Canvas control and A2UI rendering surfaces for paired nodes.",
99
"skills": ["./skills"],
1010
"contracts": {
11-
"tools": ["canvas"]
11+
"tools": ["canvas", "show_widget"]
1212
},
1313
"configContracts": {
1414
"compatibilityMigrationPaths": ["canvasHost"]

extensions/canvas/src/documents.ts

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ type CanvasDocumentCreateInput = {
3131
entrypoint?: CanvasDocumentEntrypoint;
3232
assets?: CanvasDocumentAsset[];
3333
surface?: "assistant_message" | "tool_card" | "sidebar";
34+
retentionScope?: string;
35+
/** Serve the document with a CSP sandbox header so direct opens get an opaque origin. */
36+
cspSandbox?: "scripts";
3437
};
3538

3639
type CanvasDocumentManifest = {
@@ -43,6 +46,8 @@ type CanvasDocumentManifest = {
4346
localEntrypoint?: string;
4447
externalUrl?: string;
4548
surface?: "assistant_message" | "tool_card" | "sidebar";
49+
retentionScope?: string;
50+
cspSandbox?: "scripts";
4651
assets: Array<{
4752
logicalPath: string;
4853
contentType?: string;
@@ -126,6 +131,51 @@ function resolveCanvasDocumentsDir(rootDir?: string, stateDir = resolveStateDir(
126131
return path.join(resolveCanvasRootDir(rootDir, stateDir), CANVAS_DOCUMENTS_DIR_NAME);
127132
}
128133

134+
async function pruneCanvasDocumentsForScope(params: {
135+
documentsDir: string;
136+
retentionScope: string;
137+
maxDocuments: number;
138+
}): Promise<void> {
139+
const entries = await fs.readdir(params.documentsDir, { withFileTypes: true });
140+
const scopedDocuments = (
141+
await Promise.all(
142+
entries
143+
.filter((entry) => entry.isDirectory())
144+
.map(async (entry) => {
145+
try {
146+
const manifest = JSON.parse(
147+
await fs.readFile(
148+
path.join(params.documentsDir, entry.name, "manifest.json"),
149+
"utf8",
150+
),
151+
) as { createdAt?: unknown; retentionScope?: unknown };
152+
if (
153+
manifest.retentionScope !== params.retentionScope ||
154+
typeof manifest.createdAt !== "string"
155+
) {
156+
return null;
157+
}
158+
return { id: entry.name, createdAt: manifest.createdAt };
159+
} catch {
160+
return null;
161+
}
162+
}),
163+
)
164+
).filter((entry): entry is { id: string; createdAt: string } => entry !== null);
165+
const deleteCount = Math.max(0, scopedDocuments.length - params.maxDocuments);
166+
const oldest = scopedDocuments
167+
.toSorted(
168+
(left, right) =>
169+
left.createdAt.localeCompare(right.createdAt) || left.id.localeCompare(right.id),
170+
)
171+
.slice(0, deleteCount);
172+
await Promise.all(
173+
oldest.map((entry) =>
174+
fs.rm(path.join(params.documentsDir, entry.id), { recursive: true, force: true }),
175+
),
176+
);
177+
}
178+
129179
/** Resolves the on-disk directory for one Canvas document id. */
130180
export function resolveCanvasDocumentDir(
131181
documentId: string,
@@ -299,7 +349,12 @@ async function materializeEntrypoint(
299349
/** Creates a Canvas document directory, copies assets, and writes its manifest. */
300350
export async function createCanvasDocument(
301351
input: CanvasDocumentCreateInput,
302-
options?: { stateDir?: string; workspaceDir?: string; canvasRootDir?: string },
352+
options?: {
353+
stateDir?: string;
354+
workspaceDir?: string;
355+
canvasRootDir?: string;
356+
maxDocumentsPerScope?: number;
357+
},
303358
): Promise<CanvasDocumentManifest> {
304359
const workspaceDir = options?.workspaceDir ?? process.cwd();
305360
const id = input.id?.trim() ? normalizeCanvasDocumentId(input.id) : canvasDocumentId();
@@ -320,13 +375,23 @@ export async function createCanvasDocument(
320375
? { preferredHeight: input.preferredHeight }
321376
: {}),
322377
...(input.surface ? { surface: input.surface } : {}),
378+
...(input.retentionScope ? { retentionScope: input.retentionScope } : {}),
379+
...(input.cspSandbox ? { cspSandbox: input.cspSandbox } : {}),
323380
createdAt: new Date().toISOString(),
324381
entryUrl: entry.entryUrl,
325382
...(entry.localEntrypoint ? { localEntrypoint: entry.localEntrypoint } : {}),
326383
...(entry.externalUrl ? { externalUrl: entry.externalUrl } : {}),
327384
assets,
328385
};
329386
await writeManifest(root, manifest);
387+
if (input.retentionScope && options?.maxDocumentsPerScope) {
388+
// Bounded transcript widgets cannot grow managed Canvas storage without limit.
389+
await pruneCanvasDocumentsForScope({
390+
documentsDir: resolveCanvasDocumentsDir(options.canvasRootDir, options.stateDir),
391+
retentionScope: input.retentionScope,
392+
maxDocuments: options.maxDocumentsPerScope,
393+
});
394+
}
330395
return manifest;
331396
}
332397

extensions/canvas/src/host/server.test.ts

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,52 @@ describe("canvas host", () => {
229229
}
230230
});
231231

232+
it("serves sandbox-marked documents with a CSP sandbox header and no live reload", async () => {
233+
const dir = await createCaseDir();
234+
const docDir = path.join(dir, "documents", "widget-1");
235+
await fs.mkdir(docDir, { recursive: true });
236+
await fs.writeFile(
237+
path.join(docDir, "manifest.json"),
238+
JSON.stringify({ id: "widget-1", cspSandbox: "scripts" }),
239+
"utf8",
240+
);
241+
await fs.writeFile(path.join(docDir, "index.html"), "<html><body>widget</body></html>", "utf8");
242+
const plainDir = path.join(dir, "documents", "plain-1");
243+
await fs.mkdir(plainDir, { recursive: true });
244+
await fs.writeFile(
245+
path.join(plainDir, "manifest.json"),
246+
JSON.stringify({ id: "plain-1" }),
247+
"utf8",
248+
);
249+
await fs.writeFile(
250+
path.join(plainDir, "index.html"),
251+
"<html><body>plain</body></html>",
252+
"utf8",
253+
);
254+
const handler = await createTestCanvasHostHandler(dir);
255+
256+
try {
257+
const widget = await captureHandlerResponse(
258+
handler,
259+
`${CANVAS_HOST_PATH}/documents/widget-1/index.html`,
260+
);
261+
expect(widget.status).toBe(200);
262+
// Opaque origin on direct navigation: widget script must not run as the app origin.
263+
expect(widget.headers["content-security-policy"]).toBe("sandbox allow-scripts");
264+
expect(widget.body).toContain("widget");
265+
expect(widget.body).not.toContain(CANVAS_WS_PATH);
266+
267+
const plain = await captureHandlerResponse(
268+
handler,
269+
`${CANVAS_HOST_PATH}/documents/plain-1/index.html`,
270+
);
271+
expect(plain.status).toBe(200);
272+
expect(plain.headers["content-security-policy"]).toBeUndefined();
273+
} finally {
274+
await handler.close();
275+
}
276+
});
277+
232278
it("caps live reload WebSocket inbound payloads", async () => {
233279
const dir = await createCaseDir();
234280
const constructorOptions: unknown[] = [];

0 commit comments

Comments
 (0)