You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The API proposal doesn't include support for image input to the tool and for the tool output to include images. This seems like a basic use-case we should support. Worth looking at how MCP is doing it but we'll need something web specific, allowing devs to use image elements for this as an example.
exporttypeContentBlock=|TextContent|ImageContent|AudioContent|ResourceLink|EmbeddedResource;exportinterfaceCallToolResultextendsResult{/** * A list of content objects that represent the unstructured result of the tool call. */content: ContentBlock[];
...
Images can be returned as an ImageContent or an EmbeddedResource. Both of which request the image be represented as a Base64-encoded string.
/** * An image provided to or from an LLM. */exportinterfaceImageContent{type: "image";/** * The base64-encoded image data. * * @format byte */data: string;/** * The MIME type of the image. Different providers may support different image types. */mimeType: string;/** * Optional annotations for the client. */annotations?: Annotations;/** * See [General fields: `_meta`](/specification/2025-06-18/basic/index#meta) for notes on `_meta` usage. */_meta?: {[key: string]: unknown};}
I believe most inference providers accept base64. It would nice if the browser did the conversion so the server code could be as simple as this
window.navigator.modelContext.registerTool({name: "get_product_image",description: "Get the product image from the page",inputSchema: {type: "object",properties: {productId: {type: "string"}}},handler: async({ productId })=>{constimg=document.querySelector(`#product-${productId} img`);return{content: [{type: "image",element: img}]};}});
Thanks for the pointer @MiguelsPizza! +1 to this: "It would nice if the browser did the conversion so the server code could be as simple as this". There are a few other data types to reference pixel content in Web APIs, canvas2d is a good reference for that. Documentation here.
Other than data and mimeType which will come from the image source above, MCP data structure has annotations and _meta. Those are shared types that we should discuss separately. Doesn't need to block the addition of a return type for image resources, we can always define a base structure and add to it going forward.
Related, there should likely be a way to reference the image content in a json response. If you're returning a list of products, you likely have a json with text info and placeholders for the image of each product. Does MCP already support that concept? Some overlap with #9 too.
quick revival :)
polyfill prototype exists #260 supporting both shapes; spec should bless native types per tomayac's direction, with {type:"image", data, mimeType} as the serialized fallback.
+1 to the direction here: the browser converts native image types, with {type: "image", data, mimeType} as the serialized form.
One use case this doesn't cover yet. I'm building an app where images live in server storage, not in the page, and access can be revoked mid-task. Returning inline base64 means the data can't be taken back, so we return a short-lived reference instead and check permissions on every call.
Would the group consider allowing a reference form (URL or handle, MIME type, alt text) next to inline data? It might also answer @khushalsagar's question about pointing to images from a JSON result.
Also: should untrustedContentHint cover text derived from images, like captions or OCR? That text can carry prompt injection.
Some input from the agent-host side on the open questions here.
On shape, the direction of the thread, { type: "image", data, mimeType } as the serialized form, matches MCP's image content. That's useful beyond tidiness: an agent that also talks to MCP servers can treat a page tool's image like any other tool image, with no translation step.
Size is the main practical problem. Neither spec says how large a result may be, and a screenshot or a canvas export is easily several megabytes once it's base64 in a JSON string, which the host then puts into a model's context or a request. Since the data is page-supplied we treat it as untrusted input: we check its encoded length before decoding it, skip blocks above a limit, and only look at the first few blocks. Hosts will each pick their own limits, so it would help if the spec said that hosts may cap, gave a recommended maximum, and described what a tool can do instead, which is return a reference (the thread already asks about URLs or handles with a MIME type and alt text) and let the host fetch what it needs.
On whether untrustedContentHint should cover text derived from images: yes, and the image itself too. Images can carry instructions in the pixels (text rendered into a picture), which a vision model will read, so an image from a page tool needs the same handling as returned text. If the hint is meant to cover the payload, it would be worth saying that images are part of it.
This is also related to #9, which adds outputSchema, and #282, about telling an error from a result. All three are moving towards typed results, and it would be good for them to end up as one coherent shape, ideally close to MCP's.
Activity
MiguelsPizza commentedon Oct 23, 2025
Providing some reference material for the tool result types in the MCP spec:
Included in the ToolResult object is an array of Content Blocks
Images can be returned as an ImageContent or an EmbeddedResource. Both of which request the image be represented as a Base64-encoded string.
I believe most inference providers accept base64. It would nice if the browser did the conversion so the server code could be as simple as this
Link to webmachinelearning#41
Minor formatting corrections (#42)
khushalsagar commentedon Oct 24, 2025
Thanks for the pointer @MiguelsPizza! +1 to this: "It would nice if the browser did the conversion so the server code could be as simple as this". There are a few other data types to reference pixel content in Web APIs, canvas2d is a good reference for that. Documentation here.
Other than
dataandmimeTypewhich will come from the image source above, MCP data structure hasannotationsand_meta. Those are shared types that we should discuss separately. Doesn't need to block the addition of a return type for image resources, we can always define a base structure and add to it going forward.Related, there should likely be a way to reference the image content in a json response. If you're returning a list of products, you likely have a json with text info and placeholders for the image of each product. Does MCP already support that concept? Some overlap with #9 too.
tomayac commentedon Oct 27, 2025
We should likely do what the Prompt API does here:
ImageBitmapSourceandBufferSource.AudioBuffer, againBufferSource, or plainBlob.arnabwithab commentedon Aug 31, 2026
quick revival :)
polyfill prototype exists #260 supporting both shapes; spec should bless native types per tomayac's direction, with {type:"image", data, mimeType} as the serialized fallback.
[-]Support image input/output in tools[/-][+]Support multimodal tool input/output[/+]srirammk0 commentedon Oct 5, 2026
+1 to the direction here: the browser converts native image types, with {type: "image", data, mimeType} as the serialized form.
One use case this doesn't cover yet. I'm building an app where images live in server storage, not in the page, and access can be revoked mid-task. Returning inline base64 means the data can't be taken back, so we return a short-lived reference instead and check permissions on every call.
Would the group consider allowing a reference form (URL or handle, MIME type, alt text) next to inline data? It might also answer @khushalsagar's question about pointing to images from a JSON result.
Also: should untrustedContentHint cover text derived from images, like captions or OCR? That text can carry prompt injection.
kenchris commentedon Oct 20, 2026
Some input from the agent-host side on the open questions here.
On shape, the direction of the thread,
{ type: "image", data, mimeType }as the serialized form, matches MCP's image content. That's useful beyond tidiness: an agent that also talks to MCP servers can treat a page tool's image like any other tool image, with no translation step.Size is the main practical problem. Neither spec says how large a result may be, and a screenshot or a canvas export is easily several megabytes once it's base64 in a JSON string, which the host then puts into a model's context or a request. Since the data is page-supplied we treat it as untrusted input: we check its encoded length before decoding it, skip blocks above a limit, and only look at the first few blocks. Hosts will each pick their own limits, so it would help if the spec said that hosts may cap, gave a recommended maximum, and described what a tool can do instead, which is return a reference (the thread already asks about URLs or handles with a MIME type and alt text) and let the host fetch what it needs.
On whether
untrustedContentHintshould cover text derived from images: yes, and the image itself too. Images can carry instructions in the pixels (text rendered into a picture), which a vision model will read, so an image from a page tool needs the same handling as returned text. If the hint is meant to cover the payload, it would be worth saying that images are part of it.This is also related to #9, which adds
outputSchema, and #282, about telling an error from a result. All three are moving towards typed results, and it would be good for them to end up as one coherent shape, ideally close to MCP's.