Skip to content

Commit d6649bf

Browse files
authored
WebPlatform: implement WebMCP (document.modelContext) (#4878)
* WebPlatform: implement WebMCP (document.modelContext) - document.modelContext and the ModelContext interface of the WebMCP specification: registerTool() (name/description validation, schema serialization, exposedTo, signal), getTools() (frame tree, same-origin and fromOrigins/exposedTo filtering, sorted by name), executeTool() (pending executions per traversable, cancellation, toolcancel and toolactivated events, unloading document cleanup steps), toolchange - the Tool interface of webmachinelearning/webmcp#348 (what Chromium and the WPT implement) instead of the RegisteredTool dictionary, and the outputSchema member - waitForToolsUntil() and toolsReady of webmachinelearning/webmcp#373, marked [Experimental("WEBPLATEXP0018")] - the "tools" permissions policy feature and WebPlatformFeatures.WebMcpEnabled - a host API for the application, as the browser agent: BrowsingContext.GetModelContextTools(), ExecuteModelContextToolAsync() and ModelContextToolsReady - ScriptEngine JSON hooks (JSON.stringify/JSON.parse of the realm) - WPT: run webmcp; the declarative API (explainer only) is skipped, and the subtests that expect unquoted string results are TEST BUGs (webmachinelearning/webmcp#361) * WebPlatform: address the review of WebMCP - the settlement of a tool and of a waitForToolsUntil() promise is handled by a task of the event loop (a task of the application can complete on another thread) - the realms capture JSON.parse() and JSON.stringify() when they are created, before the page can replace them - executeTool() checks the caller again after serializing the arguments (toJSON() can detach it) - Tool.Document is the document that registered the tool
1 parent 39c1e74 commit d6649bf

36 files changed

Lines changed: 2778 additions & 90 deletions

File tree

‎docs/WebPlatform/06-Scripting/01-Overview.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,3 +65,5 @@ browser, and the functions that you add.
6565
add .NET functions to the pages.
6666
- [Running untrusted pages](04-RunningUntrustedPages.md): the limits of the engine and the requests of the pages.
6767
- [WebAssembly](05-WebAssembly.md): run the WebAssembly modules of the pages with Wasmtime.
68+
- [Calling the tools of a page (WebMCP)](06-WebMcpTools.md): list the tools that a page registers with WebMCP, and
69+
execute them from .NET.
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
title: Calling the tools of a page (WebMCP)
3+
description: List the tools that a page registers with WebMCP, and execute them from .NET as the agent of a browser does.
4+
---
5+
6+
[WebMCP](https://webmachinelearning.github.io/webmcp/) lets a page expose its features as tools: JavaScript functions
7+
with a name, a natural language description and a JSON Schema of their input, registered with
8+
`document.modelContext.registerTool()`. An agent (for instance an AI model that drives the page) reads the tools and calls
9+
them instead of clicking through the page. The application plays the role of the agent: it lists the tools of a browsing
10+
context and executes them with JSON arguments.
11+
12+
<!-- ref:../../../tests/Meziantou.Framework.WebPlatform.Tests/JavaScript/ModelContextTests.cs;region=WebPlatform_WebMcp;format=md-fenced-code-block;dedent=true -->
13+
```csharp
14+
using var engine = new JavaScriptScriptEngine();
15+
using var context = new BrowsingContext(new BrowsingContextOptions { ScriptEngine = engine });
16+
await context.NavigateAsync("https://example.com/", _ => """
17+
<script>
18+
document.modelContext.registerTool({
19+
name: 'add',
20+
description: 'Adds two numbers',
21+
inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] },
22+
annotations: { readOnlyHint: true },
23+
execute: ({ a, b }) => ({ sum: a + b }),
24+
});
25+
</script>
26+
""");
27+
await context.EventLoop.RunUntilIdleAsync();
28+
29+
// The tools registered by the page and by its iframes
30+
var tool = Assert.Single(context.GetModelContextTools());
31+
Assert.Equal("add", tool.Name);
32+
Assert.Equal("Adds two numbers", tool.Description);
33+
Assert.Equal("""{"type":"object","properties":{"a":{"type":"number"},"b":{"type":"number"}},"required":["a","b"]}""", tool.InputSchemaJson);
34+
Assert.Equal(true, tool.ToolAnnotations?.ReadOnlyHint);
35+
Assert.Equal("https://example.com", tool.Origin);
36+
37+
// The arguments and the result are JSON texts. Run the event loop until the tool completes.
38+
var execution = context.ExecuteModelContextToolAsync(tool, """{"a":1,"b":2}""");
39+
await context.EventLoop.RunUntilIdleAsync();
40+
Assert.Equal("""{"sum":3}""", await execution);
41+
```
42+
<!-- endref -->
43+
44+
## Listing the tools
45+
46+
[BrowsingContext.GetModelContextTools](cref:Meziantou.Framework.WebPlatform.Dom.BrowsingContext.GetModelContextTools)
47+
returns the tools registered by the active document of the browsing context and by the documents of its iframes, in tree
48+
order, then in the order of their registration. Each [Tool](cref:Meziantou.Framework.WebPlatform.Dom.Tool) is a snapshot:
49+
its name, title, description, the JSON of its input and output schemas
50+
([InputSchemaJson](cref:Meziantou.Framework.WebPlatform.Dom.Tool.InputSchemaJson) and
51+
[OutputSchemaJson](cref:Meziantou.Framework.WebPlatform.Dom.Tool.OutputSchemaJson), null when the page gave none), its
52+
[ToolAnnotations](cref:Meziantou.Framework.WebPlatform.Dom.Tool.ToolAnnotations) (`readOnlyHint`, `consequentialHint`...),
53+
and the [Document](cref:Meziantou.Framework.WebPlatform.Dom.Tool.Document) and
54+
[Origin](cref:Meziantou.Framework.WebPlatform.Dom.Tool.Origin) that registered it. The application sees the tools of every
55+
origin of the frame tree: the `exposedTo` option of the page only restricts the other documents.
56+
57+
The list does not change when the page registers or unregisters a tool: call the method again after running the event
58+
loop. A page that registers its tools asynchronously (after fetching their configuration, for example) can call
59+
`document.modelContext.waitForToolsUntil(promise)`:
60+
[BrowsingContext.ModelContextToolsReady](cref:Meziantou.Framework.WebPlatform.Dom.BrowsingContext.ModelContextToolsReady)
61+
is false until the promises settle.
62+
63+
## Executing a tool
64+
65+
[BrowsingContext.ExecuteModelContextToolAsync](cref:Meziantou.Framework.WebPlatform.Dom.BrowsingContext.ExecuteModelContextToolAsync)
66+
takes the JSON text of the arguments (an object; `{}` by default), runs the `execute` callback of the tool in a task of
67+
the event loop, and completes with the JSON text of its result. Call it on the thread of the event loop, then run the event
68+
loop until the task completes: the callback usually returns a promise. The page sees a `toolactivated` event when the
69+
callback starts.
70+
71+
The task fails with a [DomException](cref:Meziantou.Framework.WebPlatform.Dom.DomException):
72+
73+
| Name | Reason |
74+
|---|---|
75+
| `NotFoundError` | The tool was unregistered, or its document navigated to another origin. A tool registered again with the same name is a new tool: list the tools again. |
76+
| `DataError` | The arguments are not the JSON text of an object. |
77+
| `OperationError` | The callback threw or rejected, or its result cannot be serialized to JSON. The message of the page is not given. |
78+
| `AbortError` | The document of the tool was unloaded (navigated, or its iframe removed) before the tool completed. |
79+
| `InvalidStateError` | The document of the tool was destroyed, or belongs to another frame tree. |
80+
81+
The cancellation token cancels the execution: the task is canceled, the `AbortSignal` given to the callback is aborted, and
82+
the page sees a `toolcancel` event.
83+
84+
## What the scripts can use
85+
86+
The pages use the API of the specification on `document.modelContext`, in the secure contexts (`https:`, `localhost`):
87+
`registerTool()` (with the `exposedTo` and `signal` options), `getTools()` (with `fromOrigins`), `executeTool()` (with
88+
`signal`), the `toolchange`, `toolactivated` and `toolcancel` events, and the `Tool` objects that `getTools()` returns.
89+
Their behavior follows the specification and the proposals that Chromium implements:
90+
91+
- `getTools()` returns `Tool` objects, which identify a registration of a tool
92+
([webmcp#348](https://github.com/webmachinelearning/webmcp/pull/348)): executing a tool that was unregistered fails,
93+
even when a tool with the same name was registered again.
94+
- `waitForToolsUntil()` and `toolsReady`
95+
([webmcp#373](https://github.com/webmachinelearning/webmcp/pull/373)).
96+
- `executeTool()` resolves with the JSON serialization of the result of the tool: a string is quoted, and `undefined` is
97+
`null` ([webmcp#361](https://github.com/webmachinelearning/webmcp/issues/361)).
98+
99+
The tools are restricted to the documents of the frame tree that are same origin with the document that registered them,
100+
unless the tool lists other origins in `exposedTo` and the caller lists its origin in `fromOrigins`. The iframes need the
101+
`tools` permissions policy feature (`<iframe allow="tools">`) for a cross-origin document. The declarative API of the
102+
[explainer](https://github.com/webmachinelearning/webmcp/blob/main/declarative-api-explainer.md) (the `toolname`
103+
attribute of the forms) is not implemented.
104+
105+
`new WebPlatformFeatures { WebMcpEnabled = false }` in
106+
[BrowsingContextOptions.Features](cref:Meziantou.Framework.WebPlatform.Dom.BrowsingContextOptions.Features) hides
107+
`document.modelContext` and the WebMCP interfaces from the scripts (see
108+
[WebPlatformFeatures.WebMcpEnabled](cref:Meziantou.Framework.WebPlatform.Dom.WebPlatformFeatures.WebMcpEnabled)).

‎docs/WebPlatform/10-WptCoverage/01-Overview.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,18 +48,18 @@ The features of `REQUIRES`:
4848

4949
| Group | Passing | Tests | Disabled | Pass rate |
5050
|---|--:|--:|--:|--:|
51-
| [Stable](#stable-tests) | 26,437 | 29,911 | 2,117 | 95% |
51+
| [Stable](#stable-tests) | 26,480 | 29,911 | 2,063 | 95% |
5252
| [Tentative](#tentative-tests) | 1,383 | 3,169 | 698 | 56% |
5353
| [Reftests](#reftests) | 28,250 | 29,507 | 171 | 96% |
54-
| **All** | **56,070** | **62,587** | **2,986** | **94%** |
54+
| **All** | **56,113** | **62,587** | **2,932** | **94%** |
5555

5656
## Failure reasons
5757

5858
The tests that do not pass (failing or disabled), by kind of reason. A test with several reasons counts once per kind.
5959

6060
- **BUG**: 184
61-
- **TEST BUG**: 1,019
62-
- **NOT IMPLEMENTED**: 4,334
61+
- **TEST BUG**: 1,027
62+
- **NOT IMPLEMENTED**: 4,283
6363
- **REQUIRES**: 765
6464
- `hardware`: 573
6565
- `layout`: 23
@@ -396,7 +396,7 @@ The tests that do not pass (failing or disabled), by kind of reason. A test with
396396
| [`webgl`](webgl.md) | 0 | 22 | 0 | 0% |
397397
| [`webhid`](webhid.md) | 0 | 5 | 5 | – |
398398
| [`webidl`](webidl.md) | 42 | 46 | 2 | 95% |
399-
| [`webmcp`](webmcp.md) | 0 | 79 | 79 | – |
399+
| [`webmcp`](webmcp.md) | 43 | 79 | 25 | 80% |
400400
| [`webmessaging`](webmessaging.md) | 131 | 131 | 0 | 100% |
401401
| [`webmidi`](webmidi.md) | 0 | 1 | 1 | – |
402402
| [`webnn`](webnn.md) | 0 | 192 | 192 | – |
@@ -419,7 +419,7 @@ The tests that do not pass (failing or disabled), by kind of reason. A test with
419419
| [`x-frame-options`](x-frame-options.md) | 6 | 6 | 0 | 100% |
420420
| [`xhr`](xhr.md) | 320 | 331 | 0 | 97% |
421421
| [`xml`](xml.md) | 0 | 12 | 12 | – |
422-
| **Total (343 sections)** | **26,437** | **29,911** | **2,117** | **95%** |
422+
| **Total (343 sections)** | **26,480** | **29,911** | **2,063** | **95%** |
423423

424424
## Tentative tests
425425

0 commit comments

Comments
 (0)