Status: Experimental Draft
window.inference may be injected by a browser or extension. After checking that it exists, websites call:
window.inference.request<R extends InferenceRequest>(request: R): AsyncIterable<InferenceChunkFor<R>>
window.inference.getFeatures(): InferenceFeaturesrequest is required. getFeatures reports optional capabilities (see Feature discovery). Implementations that omit getFeatures are treated as advertising none.
type ImageMediaType =
| "image/jpeg"
| "image/png"
| "image/webp"
| "image/gif";
type TextPart = {
type: "text";
text: string;
};
/**
* Image bytes (`data`) or a page-side URL (`url`), never both.
* `done.message` image parts always use `{ mediaType, data }` as a base64 string.
* On the page-facing API, `data` may be a `Blob`; injectors encode before an
* isolated-world hop. `{ url }` is resolved in the page (same CORS as the site).
*/
type ImagePart =
| {
type: "image";
mediaType: ImageMediaType;
data: string | Blob;
}
| { type: "image"; url: string };
type ContentPart = TextPart | ImagePart;
type ChatRequest = {
method: "chat";
messages: Message[];
/**
* Function tools and/or `{ type: "web_search" }`.
* Function tools only when getFeatures().toolCalling is true.
* `{ type: "web_search" }` only when getFeatures().webSearch is true.
*/
tools?: Tool[];
toolChoice?: ToolChoice;
/**
* Image generation for this turn.
* `output.images` only when getFeatures().imageOutput is true.
* Extra keys under `output` are ignored (forward compatible).
*/
output?: {
images?: boolean;
};
/** Generation preferences for this request. See InferenceOptions. */
options?: InferenceOptions;
signal?: AbortSignal;
};
/** Finite JSON values only; no cycles, sparse arrays, binary, or custom objects. */
type DecisionJsonValue =
| null | boolean | number | string
| DecisionJsonValue[]
| { [key: string]: DecisionJsonValue };
type DecisionState = string | DecisionJsonValue[] | { [key: string]: DecisionJsonValue };
/** Inline image only. String data is raw base64, never a data URL. */
type DecisionImageAttachment = {
type: "image";
mediaType: ImageMediaType;
data: string | Blob;
};
type NoulQuestion = {
type: "noul";
instructions: string;
criteria?: { true?: string; false?: string };
};
type ChoiceQuestion = {
type: "choice";
instructions: string;
/** 2–26 named choices; null means the key needs no further description. */
criteria: Record<string, string | null>;
};
type ScoreQuestion = {
type: "score";
instructions: string;
/** 2–10 ordered descriptions, indexed from zero. */
criteria: string[];
};
type DecisionQuestion = NoulQuestion | ChoiceQuestion | ScoreQuestion;
type DecideRequest = {
method: "decide";
state: DecisionState;
/** Shared native images; only advertised decisionMedia.imageMediaTypes. */
attachments?: DecisionImageAttachment[];
/**
* Nonempty questions over the same state; implementation/route limits apply.
* No dependencies between answers.
*/
questions: Record<string, DecisionQuestion>;
signal?: AbortSignal;
};
/** Exactly one page-owned source. String data is raw base64, never a data URL. */
type TranscriptionMediaSource =
| { mediaType: string; data: string; url?: never }
| { mediaType?: string; data: Blob; url?: never }
| { url: string; mediaType?: string; data?: never };
type TranscribeRequest = {
method: "transcribe";
media: TranscriptionMediaSource;
/** Structurally valid BCP 47 hint; omitted means detect. */
language?: string;
signal?: AbortSignal;
};
type SynthesisMediaType = "audio/mpeg" | "audio/wav";
type SynthesizeRequest = {
method: "synthesize";
text: string;
/** Omitted output or mediaType means MP3; WAV requires explicit discovery. */
output?: { mediaType?: SynthesisMediaType };
signal?: AbortSignal;
};
type InferenceRequest = ChatRequest | DecideRequest | TranscribeRequest | SynthesizeRequest;
type InferenceMethod = InferenceRequest["method"];
type InferenceOptions = {
/**
* Preferred reasoning / thinking effort.
* Distinct from assistant `message.reasoning` (output text).
*/
reasoningEffort?: ReasoningEffort;
/**
* Sampling temperature in `[0, 2]`.
* Omitted means the implementation / provider default.
*/
temperature?: number;
};
/** Omitted or `"auto"` means the implementation / provider default. */
type ReasoningEffort = "auto" | "none" | "low" | "medium" | "high";
type Message =
| {
role: "system";
content: string;
}
| {
role: "user";
content: string | ContentPart[];
}
| {
role: "assistant";
content: string | ContentPart[] | null;
/** Model reasoning / chain-of-thought, when the provider exposes it. */
reasoning?: string;
toolCalls?: ToolCall[];
}
| {
role: "tool";
toolCallId: string;
content: string;
};
type Usage = {
inputTokens?: number;
outputTokens?: number;
};
type ChatChunk =
| { type: "accepted" }
| { type: "reasoning_delta"; content: string }
| { type: "delta"; content: string }
| { type: "done"; model: string; message: Message; usage?: Usage };
type NoulAnswer = { type: "noul"; noul: number };
type ChoiceAnswer = {
type: "choice";
choice: string;
probabilities: Record<string, number>;
confidence: number;
};
type ScoreAnswer = {
type: "score";
score: number;
legend: Record<string, string>;
probabilities: Record<string, number>;
confidence: number;
};
type RefusalAnswer = { type: "refusal" };
type DecisionAnswer = NoulAnswer | ChoiceAnswer | ScoreAnswer | RefusalAnswer;
type DecisionChunk =
| { type: "accepted" }
| { type: "done"; model: string; answers: Record<string, DecisionAnswer>; usage?: Usage };
type TranscriptionUsage = { inputSeconds?: number };
type SynthesisUsage = {
inputCharacters?: number;
outputSeconds?: number;
};
type TranscriptionChunk =
| { type: "accepted" }
| { type: "delta"; content: string }
| {
type: "done";
model: string;
transcript: { text: string; language?: string };
usage?: TranscriptionUsage;
};
type SynthesisChunk =
| { type: "accepted" }
| { type: "audio_delta"; mediaType: SynthesisMediaType; data: Uint8Array }
| {
type: "done";
model: string;
audio: { mediaType: SynthesisMediaType; byteLength: number };
usage?: SynthesisUsage;
};
type InferenceChunk = ChatChunk | DecisionChunk | TranscriptionChunk | SynthesisChunk;
/** Distributes over request unions while preserving concrete method inference. */
type InferenceChunkFor<R extends InferenceRequest> =
R extends ChatRequest ? ChatChunk :
R extends DecideRequest ? DecisionChunk :
R extends TranscribeRequest ? TranscriptionChunk :
R extends SynthesizeRequest ? SynthesisChunk : never;
type InferenceDoneFor<R extends InferenceRequest> =
Extract<InferenceChunkFor<R>, { type: "done" }>;
type ChatDoneChunk = InferenceDoneFor<ChatRequest>;
type DecisionDoneChunk = InferenceDoneFor<DecideRequest>;
type TranscriptionDoneChunk = InferenceDoneFor<TranscribeRequest>;
type SynthesisDoneChunk = InferenceDoneFor<SynthesizeRequest>;
type TranscriptionInputConstraint = {
/** Normalized MIME essence; no wildcards. */
mediaType: string;
containers?: string[];
audioCodecs?: string[];
};
type InferenceFeatures = {
/** Chat is required. Absent or false optional methods mean unsupported. */
methods?: {
/** All three decision primitives and refusals; request size limits apply. */
decide?: boolean;
transcribe?: { acceptedMedia: TranscriptionInputConstraint[] };
/** true means MP3 only; an object declares exactly its nonempty format list. */
synthesize?: boolean | { outputMediaTypes: SynthesisMediaType[] };
};
/** Independent of chat imageInput; requires methods.decide === true. */
decisionMedia?: { imageMediaTypes: ImageMediaType[] };
/**
* Implementation accepts function tools, toolChoice, and tool messages
* on request. Absent or false means unsupported.
*/
toolCalling?: boolean;
/**
* Implementation accepts `{ type: "web_search" }` in `tools`.
* Absent or false means unsupported. Independent of `toolCalling`.
*/
webSearch?: boolean;
/**
* Implementation accepts ImageParts in `messages` (user uploads and
* round-tripped assistant images). Absent or false means unsupported.
* Independent of `imageOutput`.
*/
imageInput?: boolean;
/**
* Implementation accepts `output.images`; `done.message` may contain
* ImageParts. Absent or false means unsupported. Independent of `imageInput`.
*/
imageOutput?: boolean;
/**
* Which InferenceOptions keys this implementation accepts.
* Absent keys (and an absent options object) mean ignore those fields.
*/
options?: {
reasoningEffort?: boolean;
temperature?: boolean;
};
};
type Tool =
| {
type: "function";
function: {
name: string;
description?: string;
/** JSON Schema object for the function arguments. */
parameters?: { [key: string]: unknown };
};
}
| { type: "web_search" };
type ToolChoice =
| "auto"
| "none"
| "required"
| { type: "function"; function: { name: string } };
type ToolCall = {
id: string;
type: "function";
function: {
name: string;
/** JSON-encoded argument object. */
arguments: string;
};
};
type InferenceErrorCode =
| "permission_denied"
| "invalid_request"
| "unavailable"
| "provider_error"
| "aborted";
type InferenceError = Error & {
code: InferenceErrorCode;
};
type Inference = {
request<R extends InferenceRequest>(request: R): AsyncIterable<InferenceChunkFor<R>>;
getFeatures?(): InferenceFeatures;
};
/** Compatible chat-only alias; never widened to the all-method done union. */
type DoneChunk = ChatDoneChunk;InferenceError is an Error with a code property. Across extension isolated worlds, implementations may reconstruct errors from a serializable { name, message, code } shape rather than preserving a subclass. Applications should check error.code, not instanceof.
Text chat (method: "chat" with system / user / assistant string messages) is required. Typed decisions (method: "decide"), bounded transcription (method: "transcribe") and bounded speech synthesis (method: "synthesize"), tool calling, hosted web search, image input/output, and request options (for example reasoningEffort, temperature) are optional; implementations advertise them with getFeatures. String content on user and assistant messages stays valid so existing apps do not break.
The method determines the chunk union: chat yields ChatChunk, decisions yield
DecisionChunk, transcription yields TranscriptionChunk, and
synthesis yields SynthesisChunk. The conditional mapping distributes over an
InferenceRequest union. Equivalent overloads must also accept a union-typed
request. ChatRequest, ChatChunk, and ChatDoneChunk retain the narrow chat
contract; DoneChunk remains an alias of ChatDoneChunk.
for await (const chunk of window.inference.request({
method: "chat",
messages: [{ role: "user", content: "Hello" }],
})) {
if (chunk.type === "accepted") {
// permission resolved; provider call may begin
} else if (chunk.type === "reasoning_delta") {
// append chunk.content to a reasoning UI (optional)
} else if (chunk.type === "delta") {
// append chunk.content to the reply UI
} else if (chunk.type === "done") {
// final message / usage; message.reasoning is set when reasoning was streamed
}
}- A request begins when the application starts iterating. The extension obtains permission for the calling origin before sending any content to a provider, prompting the user unless persistent permission already exists.
- The user chooses the provider and model, or the extension uses the route previously saved for that origin and method. Synthesis also binds the user-selected voice; pages cannot select it.
- API keys never leave the extension. Applications never see them.
- If a request fails, iteration throws an
InferenceError. A failed request does not yield adonechunk. requestyields exactly oneacceptedchunk after the origin is permitted and the request is cleared to call a provider — including when a persistent grant already exists and no prompt is shown.accepteddoes not mean the user clicked Allow in a UI; applications must not assume a prompt occurred. Failures during permission or preflight do not yieldaccepted.- After
accepted, a successful request yields method-specific chunks, then exactly onedonechunk. No chunks followdone. Chat yields zero or morereasoning_deltaand/ordeltachunks (in any order); the chat-specific rules below apply tomethod: "chat". Decisions follow Typed decisions; speech follows Bounded speech. - Concatenating every
delta.contentproducesdone.message.contentwhen the assistant reply is text. Reasoning is not included incontent. On a tool turn with nodeltachunks,done.message.contentmay benullor""; applications must treat both as no text reply. When the reply includes images,done.message.contentis aContentPart[]; concatenating everydelta.contentproduces the concatenation of thetype: "text"parts, in order. Image parts appear only ondone(noimage_deltain this draft). An image-only reply usescontent: [{ type: "image", ... }, ...]with zerodeltas — do not usenullto mean “here is a PNG.” - Concatenating every
reasoning_delta.contentproducesdone.message.reasoningwhen the provider exposed reasoning. If there were noreasoning_deltachunks,done.message.reasoningis omitted. - Providers or models that do not expose reasoning yield no
reasoning_deltachunks and omitmessage.reasoning. Applications must treat reasoning as optional. - Providers that do not stream may yield no
reasoning_deltaordeltachunks and only a finaldone(withmessage.contentand optionalmessage.reasoningand/ormessage.toolCalls). - When sending prior assistant turns back in
messages, applications may includereasoningif they received it. Implementations map it to the provider when the provider supports round-tripping reasoning; otherwise they may ignore it. Applications must not rely on every provider consuming prior reasoning. optionsholds generation preferences (see Request options). They do not change the stream shape: applications must still treatreasoning_deltaandmessage.reasoningas optional.- Aborting
signal, closing the page, or navigating it aborts an active request with theabortederror code. When the document is unloading, the page may be unable to observe that rejection; implementations must still cancel in-flight provider work.
getFeatures is synchronous. It must not prompt, must not require permission, and must not perform network I/O. It returns a snapshot of API surface this implementation supports, not whether the user's current provider or model will honor a capability, and not whether the origin has a permission grant.
Applications must feature-detect. Older injectors may omit the method:
const features = window.inference.getFeatures?.() ?? {};
if (features.methods?.decide) {
// decide returns typed answers over a shared JSON state
}
if (features.methods?.decide === true && features.decisionMedia?.imageMediaTypes?.includes("image/png")) {
// decide also accepts shared inline PNG attachments
}
if (features.methods?.transcribe) {
// transcribe accepts the advertised acceptedMedia constraints
}
if (features.methods?.synthesize) {
// synthesize returns encoded audio bytes; the page controls format and playback
}
if (features.toolCalling) {
// request accepts function tools / toolChoice / tool messages
}
if (features.webSearch) {
// request accepts tools: [{ type: "web_search" }, ...]
}
if (features.imageInput) {
// messages may include ImageParts
}
if (features.imageOutput) {
// request.output.images accepted; done.message may include ImageParts
}
if (features.options?.reasoningEffort) {
// request.options.reasoningEffort accepted; implementation will try to map it
}
if (features.options?.temperature) {
// request.options.temperature accepted; implementation will try to map it
}Rules:
-
toolCalling: truemeansrequestaccepts function tools,toolChoice, assistanttoolCalls, androle: "tool"messages. It does not mean the selected model can call functions, and it does not advertise{ type: "web_search" }. -
webSearch: truemeansrequestaccepts{ type: "web_search" }intools. It does not mean the selected provider can search, and not whether the origin has a grant.webSearchandtoolCallingare independent: a search-onlytoolsarray must not requiretoolCalling; function tools must not requirewebSearch. -
options.reasoningEffort: truemeansrequest.options.reasoningEffortis accepted and the implementation attempts to map it to the provider. It does not mean the selected model supports adjustable thinking. Advertise options per key; a bareoptions: {}advertises none. The same per-key rule applies tooptions.temperatureand any lateroptionskeys. -
An absent key and
falseboth mean unsupported. Applications must ignore unknown keys (including unknown keys under chatoptionsand chatoutput) so later capabilities can be added without breaking callers. Unknown keys already allow a later draft to add a nested object (for example for MCP) without breaking callers that checkfeatures.webSearch. -
The result must not include provider name, model id, or other user or account identity.
-
Implementations that do not support tool calling must reject function tools,
toolChoicevalues other than"auto"and"none",role: "tool"messages, and assistanttoolCallswithinvalid_request— including implementations that omitgetFeatures. A search-onlytoolsarray ([{ type: "web_search" }]) must not be rejected solely for lackingtoolCalling."none"and"auto"remain valid on a search-only request so the page can suppress hosted search withouttoolCalling. -
Implementations that do not support hosted web search must reject
{ type: "web_search" }withinvalid_request— including implementations that omitgetFeatures. Function tools must not be rejected solely for lackingwebSearch. Same reject rule as function tools, not the ignore rule used foroptions. -
If
toolscontains a kind that is not advertised, or an unknowntools[].type, the whole request isinvalid_request. Do not silently drop hosted search or function tools. Extra keys on{ type: "web_search" }are ignored (forward compatible). -
Implementations must ignore unsupported
optionskeys (must not reject the request solely for including them). Applications may send future keys for forward compatibility; they have no effect until advertised. -
Advertising
toolCallingdoes not guarantee that the user's provider or model will emittoolCalls. The model may ignore tools and reply in text. Applications must handle a text-onlydoneeven when they offered tools. Implementations must not reject a well-formed tools request solely because the current model is weak at function calling. -
Advertising
webSearchdoes not guarantee that the model will actually search, only that the implementation will enable search for providers that can honor it. Applications must still handle a textdone. If the request includes{ type: "web_search" }and the currently selected provider cannot honor it (or required extra credentials are missing), do not complete a normal chat reply — see Hosted web search. -
Advertising an
optionskey does not guarantee that the user's provider or model will honor that preference. Implementations map best-effort and must not fail a well-formed request solely because the current model cannot apply the option. -
imageInput: truemeansrequestacceptsImageParts inmessages(onuserandassistantroles). It does not mean the selected model can see images, and not whether the origin has a grant. Independent ofimageOutput. -
imageOutput: truemeansrequestacceptsoutput.imagesanddone.messagemay containImageParts. It does not mean the selected model will draw, and not whether the origin has a grant. Independent ofimageInput. AdvertisingimageOutputdoes not mean the implementation accepts round-tripped assistant image parts withoutimageInput. -
Implementations that do not support image input must reject
ImageParts inmessageswithinvalid_request— including implementations that omitgetFeatures. Do not drop image parts and answer as if they were seen. Same reject rule as function tools, not the ignore rule used foroptions. -
Implementations that do not support image output must reject
output.images: truewithinvalid_request— including implementations that omitgetFeatures. Extra keys underoutputare ignored (forward compatible). If the page does not setoutput.images: true, implementations must not putImageParts ondone.message(text only, as today). -
Advertising
imageInput/imageOutputdoes not guarantee that the user's provider or model can see or draw. Applications must still handle a text-onlydonewhen they setoutput.images: true. If the request includes image parts and/oroutput.images: trueand the currently selected provider cannot honor that, do not complete a normal chat reply — see Images. -
methods.decide: trueadvertises all three typed decision primitives and per-question refusals, not unlimited capacity or a guarantee that a selected route accepts any particular request size.methods.transcribeadvertises bounded transcription and must contain at least oneacceptedMediaconstraint.methods.synthesize: trueis the legacy MP3-only advertisement; an object declares exactly its nonemptyoutputMediaTypeslist, with every entry aSynthesisMediaType. A WAV-only implementation may advertise{ outputMediaTypes: ["audio/wav"] }.falseor missing entries mean unsupported; chat remains required without amethods.chatentry. Optional methods are independent of one another and of chat capability flags. -
Unknown or unadvertised methods are
invalid_request. A well-formed advertised method that the selected provider/model/credentials cannot honor isunavailable, except that decision resource-limit violations areinvalid_request; do not substitute chat, omit questions, drop media, or silently change the approved route. -
Transcription constraints use normalized MIME essences and may include lowercase container and audio codec identifiers. Omitted narrower constraints mean no such preflight filter, not verified support for every codec. Speech discovery may describe the identity-free union of possible routes; approval must filter to matching input constraints or requested synthesis formats. Advertising a synthesis format does not guarantee the currently approved provider/model/voice route can produce it. No provider/model/voice identities or account state belong in discovery.
-
decisionMedia.imageMediaTypesdeclares exactly the image MIME types accepted in decisionattachments. When present, it must be a nonempty, duplicate-free list of canonicalImageMediaTypevalues andmethods.decidemust betrue. MissingdecisionMediameans text/JSON decisions only; an empty or malformed list advertises no media support.methods.decideremains a boolean. This capability is independent of chatimageInput/imageOutput, speech, and grants. It may describe an identity-free union of implemented routes; it must not change based on the selected model, credentials, permission, or account. Unadvertised attachment types areinvalid_request; advertised types unavailable on the approved route areunavailable. See Decision media for validation, route checks, and consent.
options is a bag of generation preferences for a chat call — not model selection (the user still chooses provider and model). This draft defines reasoningEffort and temperature; later drafts may add further keys under the same object. Applications must ignore unknown keys; implementations must ignore keys they do not advertise. Other methods have closed request shapes and do not accept options in this draft.
Lets applications prefer less or more model reasoning / chain-of-thought (for example "none" or "low" for translation or autocomplete; "high" for harder multi-step tasks). It controls generation, not merely whether the page displays reasoning: omitting UI for reasoning_delta does not reduce latency or cost if the model still thinks.
This is distinct from assistant message.reasoning / reasoning_delta, which are optional outputs when a provider exposes chain-of-thought text.
- When omitted, behavior is
"auto": the implementation uses its default (typically the provider or model default). "none"asks to disable or minimize thinking when the provider supports that."low","medium", and"high"ask for increasing effort when the provider exposes an effort or budget control. Implementations map these onto provider-specific parameters (for example effort enums or token budgets); exact mapping is implementation-defined.- When
getFeatures().options?.reasoningEffortis true, values outsideReasoningEffortareinvalid_request. - When that feature key is absent or false,
options.reasoningEffortis ignored (see Feature discovery).
Applications that care about latency or cost for simple tasks should pass "none" or "low" when support is advertised. Applications must not assume reasoning output disappears, or that every model becomes non-thinking, solely because they requested "none".
Lets applications prefer less or more sampling randomness (for example low values for translation or extraction; higher values for brainstorming). Orthogonal to reasoningEffort: temperature affects token sampling; reasoning effort affects thinking budget / chain-of-thought.
- When omitted, the implementation uses its default (typically the provider or model default).
- When present, the value must be a finite number in the closed interval
[0, 2](OpenAI-style scale). Implementations map or clamp onto provider-specific ranges when a provider uses a different scale (for example[0, 1]). - When
getFeatures().options?.temperatureis true, non-finite values or values outside[0, 2]areinvalid_request. - When that feature key is absent or false,
options.temperatureis ignored (see Feature discovery).
Applications must not assume every model honors temperature exactly; advertising means the implementation will attempt to pass it through.
options alone does not require a new permission prompt and does not widen a persistent grant. Implementations may let the user override or clamp values such as reasoningEffort or temperature in extension settings (or optionally in the approval UI). Override and clamp controls are optional extension UX — this draft does not require them on the permission prompt or elsewhere. Consent remains origin + provider/model (+ tools and image input/output when present).
Function tools are defined and executed by the page. The implementation relays JSON schemas, toolCalls, and role: "tool" results; it must not run application tool handlers or widen host permissions in order to execute tools.
This draft specifies function tools and hosted web search (see Hosted web search). Other hosted tool types (for example MCP) are out of scope. The implementation is not an agent runtime: the page owns any multi-round function-tool loop (send tools → receive toolCalls → execute → send role: "tool" results → repeat). Hosted search turns stay hidden from the page.
tools, when present, must be a non-empty array of function tools and/or{ type: "web_search" }. Mixed arrays are allowed only when bothtoolCallingandwebSearchare advertised; if the array contains a kind that is not advertised, the whole request isinvalid_request.parameterson a function tool is a JSON Schema object for the function arguments.{ type: "web_search" }has nofunctionand no parameters in this draft.toolChoiceremains a function-tool control. When omitted andtoolsis present, it defaults to"auto"(the model may reply in text or call function tools)."none"suppresses function calls and hosted search."required"asks for at least one function call; a function object forces that function."auto","required", and a named function do not force a web search. There is no{ type: "web_search" }variant oftoolChoicein this draft.role: "tool"messages must includetoolCallId(theidfrom the correspondingToolCall) and stringcontent(usually JSON text). They must not includetoolCallsorreasoning.- Assistant messages may include
toolCalls(non-empty when present).contentmay benullwhen the turn is tool-only.toolCalls[].function.argumentsis a JSON string, not a parsed object. systemandusermessages must not includetoolCallsortoolCallId.systemcontentis always a string.usercontentmay be a string or aContentPart[](see Images).
Streaming is unchanged: accepted → optional reasoning_delta / delta → done. Tool calls are not streamed as separate chunk types. When the model ends on tools, done.message is an assistant message with toolCalls (and content often null or empty).
When continuing a tool turn, the application appends that assistant message (including toolCalls and reasoning if present), then one role: "tool" message per call, then calls request again.
const features = window.inference.getFeatures?.() ?? {};
if (!features.toolCalling) {
throw new Error("This IPA implementation does not support tool calling.");
}
const tools = [
{
type: "function",
function: {
name: "get_weather",
description: "Get the current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
];
const messages = [
{ role: "user", content: "What's the weather in Austin?" },
];
let done;
for await (const chunk of window.inference.request({
method: "chat",
messages,
tools,
})) {
if (chunk.type === "done") done = chunk;
}
if (done.message.role === "assistant" && done.message.toolCalls?.length) {
messages.push({
role: "assistant",
content: done.message.content,
toolCalls: done.message.toolCalls,
...(done.message.reasoning ? { reasoning: done.message.reasoning } : {}),
});
for (const call of done.message.toolCalls) {
const args = JSON.parse(call.function.arguments);
const result =
call.function.name === "get_weather"
? { city: args.city, tempC: 22 }
: { error: "unknown tool" };
messages.push({
role: "tool",
toolCallId: call.id,
content: JSON.stringify(result),
});
}
for await (const chunk of window.inference.request({
method: "chat",
messages,
tools,
})) {
if (chunk.type === "delta") {
// append chunk.content
}
}
}The loop above is application code, not part of window.inference. A non-normative helper library (ipa-tools) provides TypeScript types, a complete drain helper, and a runTools loop; implementations are not required to ship it.
A persistent allow grant for text chat does not cover a later request that includes tools, or a wider tool set than the grant records. A function-tools grant does not cover { type: "web_search" }, and a hosted-search grant does not cover function tools. Implementations must prompt again in those cases, including when a persistent grant already exists for the origin.
When a request includes function tools, the permission UI must list the function names so the user can see what the site is authorizing the model to request. Showing descriptions is optional. Hosted search has its own disclosure (see Hosted web search). Request-content preview remains optional (see Security).
Implementations may treat a follow-up that only appends role: "tool" results for the just-approved assistant toolCalls as the same permission episode, so a multi-round tool loop does not re-prompt on every round. Adding or removing { type: "web_search" } is a wider tool set and needs a new prompt (or a covering grant). A new user turn, a new or wider tools list, or a history that is not a continuation of that episode requires a new prompt (or a covering persistent grant). Those follow-ups often have no second preview: tool result bodies can reach the provider without appearing in the Allow UI.
Applications SHOULD make the data a tool will return to the provider inspectable before the user allows inference. Do not rely on a later role: "tool" message for that disclosure. Practical options (any one can suffice):
- Include the data, or a faithful truncated copy, in the first
messagesso an implementation preview can show it. - Name the class of data in the tool
description(for example “returns the user’s currently loaded social posts”). - Disclose it in the page UI next to the action that starts
request.
This is a SHOULD, not a MUST. Live or huge tool results cannot always be inlined on the first turn. The requirement is honesty about what will be sent, not stuffing every payload into messages. Applications must not assume the permission UI reveals message content or later tool-result rounds.
Hosted web search is not page-executed. The implementation (or the selected provider) runs it. The page must not see toolCalls / role: "tool" turns for web_search, and must not implement execute.web_search.
getFeatures().webSearch === true means the implementation accepts the type, not that every provider will search, and not that the origin has a grant.
const features = window.inference.getFeatures?.() ?? {};
if (features.webSearch) {
for await (const chunk of window.inference.request({
method: "chat",
messages: [{ role: "user", content: "What's the weather in New York City today?" }],
tools: [{ type: "web_search" }],
})) {
if (chunk.type === "delta") {
// append chunk.content — search is folded into the assistant text
}
}
}Mixed arrays are allowed when both flags are advertised:
tools: [
{ type: "web_search" },
{
type: "function",
function: { name: "get_weather", parameters: { type: "object" } },
},
]Stream shape is unchanged: accepted → optional reasoning_delta / delta → done. Search is folded into the assistant text. Citations, source lists, and search-status chunks are out of scope; implementations may put links in message.content as ordinary text.
Who runs the search is an implementation detail (provider-hosted vs extension-executed, including a hidden local loop such as Ollama → ollama.com), as long as the page contract is the same.
"none" means “do not use tools this turn.” Hosted search is a tool, so "none" suppresses it as well as function calls. Pages that keep { type: "web_search" } in tools can still disable it for a given turn without rebuilding the array. Implementations must omit or disable hosted search for that request (not warn-and-continue, and not run a hidden search loop).
"auto" / "required" / a named function do not force a web search. "required" and a named function without toolCalling (or without any function tool in tools) are invalid_request.
A chat-only persistent grant does not cover { type: "web_search" }. A function-tools grant does not cover it either. Hosted search is a distinct identity in the grant.
When tools includes { type: "web_search" }, the permission UI must disclose that the provider or implementation will query the public web (and may fetch result pages). Listing a clear hosted-search label is required; showing extra description is optional.
Follow-up rounds that only append role: "tool" results for page function calls stay in the existing tools episode. Adding or removing { type: "web_search" } is a wider tool set and needs a new prompt (or a covering grant).
Do not warn-and-continue: a successful done after the page asked for search, with no search, is a silent capability lie.
If the request includes { type: "web_search" } and the currently selected provider cannot honor it (or required extra credentials are missing), do not complete a normal chat reply. Implementations should disable Allow in the permission UI, or fail the request with unavailable.
Use unavailable (not invalid_request) for this case:
{ type: "web_search" }on an implementation that advertisedwebSearchis a well-formed request. The failure is that this provider or credential set cannot honor it right now — same bucket as a missing key or a provider that is not ready.invalid_requeststays the code for unadvertised{ type: "web_search" }, unknowntools[].type, and other malformed requests. Using it for a provider mismatch would teach apps that the tool shape itself was wrong.
Prefer disabling Allow so the user never hits the error; if the request still proceeds, throw unavailable. Advertising webSearch does not guarantee the model will actually search, only that the implementation will enable search for providers that can. Applications must still handle a text done.
Vision and image generation stay on method: "chat". One ImagePart type is used in messages and on done.message. String content remains valid. Do not hang image generation under options (unsupported options are ignored; silently not generating is the same class of lie as dropping { type: "web_search" }). Do not add a page-facing { type: "image_generation" } tool in this draft; implementations may map output.images to a provider tool internally.
getFeatures().imageInput / imageOutput report API surface, not whether the current model can see or draw, and not whether the origin has a grant. The flags are independent, like toolCalling / webSearch.
const features = window.inference.getFeatures?.() ?? {};
// Vision Q&A (image input only)
if (features.imageInput) {
for await (const chunk of window.inference.request({
method: "chat",
messages: [
{
role: "user",
content: [
{ type: "text", text: "What is in this photo?" },
{ type: "image", url: "https://example.com/photo.png" },
],
},
],
})) {
if (chunk.type === "delta") {
// append chunk.content
}
}
}
// One-shot generate (image output only)
if (features.imageOutput) {
for await (const chunk of window.inference.request({
method: "chat",
messages: [{ role: "user", content: "a red fox sticker" }],
output: { images: true },
})) {
if (chunk.type === "done") {
// done.message.content may be a ContentPart[] with ImageParts,
// or text only if the model did not draw
}
}
}Edit / iterate is both in one request: image parts in messages and output.images: true. Round-trip like reasoning: if the next turn should see the picture, send that assistant message back, image parts included. imageOutput does not imply the implementation will accept those round-tripped parts without imageInput.
A non-normative generateImage helper can live in ipa-tools later without a spec change.
- Portable form:
{ type: "image", mediaType, data }wheredatais raw base64 (not adata:URL). This is whatdone.messageuses. - Blob: On the page-facing API,
datamay be aBlob. Browser injectors must acceptBloband encode to base64 before crossing into an isolated world or service worker. { url }: Browser implementations that inject into a document SHOULD accept{ type: "image", url }on user and assistant parts and MUST resolve it in the page (same CORS as the site) to{ mediaType, data }before the provider or a privileged extension world sees the request. InfermediaTypefrom the response (Content-Type/Blob.type); do not require the page to pass it. Local models stay offline — they still receive bytes. Non-page IPA (for example Node) MAY reject{ url }withinvalid_request. Fetch, CORS, or network failure isinvalid_request, not a silent drop.fetch()CORS is stricter than<img src>; prefer Blob/base64 when the page already has bytes.- XOR: A part must have
dataorurl, not both. Both or neither isinvalid_request. - MIME:
image/jpeg,image/png,image/webp,image/gif. Implementations MAY normalizeimage/jpgtoimage/jpeg. Other MIME types areinvalid_request. - Size: Implementation-defined. Fail closed when the provider or transport rejects. This draft does not set a portable byte cap.
- No host fetch: The implementation must not fetch image URLs itself in order to honor
{ url }. Page-side resolve keeps local models offline and avoids extra host URL-fetch permissions. - Image parts are not allowed on
systemortoolmessages.
Stream shape stays text-shaped: accepted → optional reasoning_delta / delta → done. No image_delta in this draft (same as no streaming toolCalls).
- Text-only reply:
contentis a string; concatenatingdeltas produces it. - Reply with images:
contentis aContentPart[]; concatenatingdeltas produces the concatenation of thetype: "text"parts, in order. Image parts appear only ondone. - Image-only reply:
contentis[{ type: "image", ... }, ...], zerodeltas. Keepnull/""as “no text” for tool-only turns; do not usenullfor “here is a PNG.”
A persistent chat grant covers neither image parts in messages nor output.images. imageInput and imageOutput are distinct grant identities (photos leaving the page vs paying for / receiving generated media), sibling to tools — not stuffed into the tools fingerprint. A request that includes image parts and output.images needs both.
Permission UI must disclose each. Thumbnails stay optional like today’s content preview.
Do not warn-and-continue: a successful done after the page sent image parts, as if they were seen, is a silent capability lie. Same for output.images: true with no generation path.
If the request includes image parts and/or output.images: true and the currently selected provider cannot honor that (or required extra credentials are missing), do not complete a normal chat reply. Implementations should disable Allow in the permission UI, or fail the request with unavailable.
Use unavailable (not invalid_request) for this case:
- Image parts /
output.imageson an implementation that advertised the matching flag is a well-formed request. The failure is that this provider or model cannot honor it right now — same bucket as hosted search. invalid_requeststays the code for unadvertised image parts /output.images, malformed parts, and other malformed requests.
Prefer disabling Allow so the user never hits the error; if the request still proceeds, throw unavailable. Advertising imageOutput does not mean the model will draw. Applications must still handle a text-only done.
method: "decide" evaluates a shared state and optional native image
attachments against named, typed questions.
It returns probabilities and bounded decisions for application code, rather
than a chat message. The vocabulary follows the TypeSafe primitives
and can be mapped to native decision APIs such as Ollama System One.
Provider protocols, endpoint names, and model catalogs remain implementation details.
Native image mappings are also possible with Gemini Decisions
and OpenAI Decisions.
This does not advertise every modality those providers accept.
Feature-detect getFeatures().methods?.decide === true before use. This flag
describes API support, independently of credentials, installed models, or site
approval. It covers noul, choice, and score, not unlimited capacity or a
guarantee that a selected route accepts any particular request size. There are
no per-provider, per-primitive, or limits-discovery flags in this draft.
Optional image support is discovered separately through decisionMedia.imageMediaTypes.
The request has only method, state, questions, and optional attachments
and signal. Reject unknown top-level, attachment, or question fields with invalid_request, including
page-selected provider, model, keep_alive, messages, tools, output,
options, or stream. The user chooses the decision provider and model;
implementations may offer different defaults from chat.
statemust be a non-whitespace string, a plain JSON object, or an array. Nested JSON values may include strings, booleans, null, and finite numbers. Empty objects and arrays are valid state; top-level null, numbers, and booleans are not. Reject cycles, undefined values, sparse arrays, non-finite numbers, functions, symbols, BigInts, binary objects, and custom objects such asDate,Map, or instances with custom serialization. Do not silently coerce or drop non-JSON values. Shared references without cycles are allowed.questionsmust be a plain object with at least one own entry. Every id must be a non-whitespace string; preserve ids exactly indone.answers. Each question hastypeand non-whitespace stringinstructions, pluscriteriaas below. Ids correlate answers; put the actual question ininstructions.- All questions are evaluated against the same state and all attachments independently. An answer cannot refer to another answer in the same request. Applications make a later request when a decision depends on an earlier result.
- Implementations must document and enforce conservative resource limits,
including maximum question counts and payload sizes. IPA fixes no universal
question-count or payload-size maximum. Document how payload size is measured
(for example, UTF-8 bytes of
JSON.stringify({ state, questions })) and which fields it includes. Count media separately as specified in Decision media, rather than charging base64 text against the JSON-state allowance. Selected providers/models may impose smaller limits. Reject requests exceeding any applicable limit withinvalid_requestand a useful explanation identifying the exceeded limit, including its value when known. Enforce locally known limits before provider inference I/O, and before approval whenever possible; check route-specific limits once the route is selected. When a limit cannot be checked locally, translate upstream limit rejections toinvalid_requestwith a useful explanation and yield nodone. - Reject deeper than 64 levels, counting the
{ state, questions, attachments }wrapper as depth 0 and each object-property or array-element descent as one level, including primitive leaves. - On first iteration, validate and snapshot
state,questions, andattachmentsbefore approval or provider I/O. Later page mutations must not alter the approved request.signalremains a cancellation handle and is never serialized. Implementations must not silently truncate questions or state, or automatically split one decision request into multiple provider calls to fit a limit. Applications may explicitly submit multiple requests when appropriate. URLs, media-shaped objects, or image-like strings inside state are ordinary data; the implementation must not fetch them or interpret them as media attachments.
| Question type | criteria |
Answer |
|---|---|---|
noul |
Optional object with only "true" and/or "false" string descriptions; either may be omitted |
{ type: "noul", noul }, where noul is the probability of yes in [0, 1] |
choice |
Object with 2–26 non-whitespace option keys and string or null descriptions; null means the key needs no extra description | { type: "choice", choice, probabilities, confidence } |
score |
Array of 2–10 string level descriptions, ordered from index 0 | { type: "score", score, legend, probabilities, confidence } |
This draft deliberately bounds question instructions and descriptions to strings. Structured instructions/criteria, per-question attachments, and additional primitives supported by some native APIs are outside this envelope.
attachments is optional. Omission and [] both mean a text/JSON-only request
and require no media capability or media grant. If present it must be a dense
array of plain DecisionImageAttachment objects with exactly type,
mediaType, and data. Every attachment is shared evidence available to every
question, in array order, alongside the complete state. Instructions can refer
to images by their zero-based attachment index; no per-question or provider ids
are added. State remains required, including for image-only evidence ({} is
valid). Binary objects are still forbidden inside state.
typemust be"image".mediaTypemust be a canonical"image/jpeg","image/png","image/webp", or"image/gif"and explicitly advertised indecisionMedia.imageMediaTypes. No MIME parameters, aliases, or wildcards.datais either a nonempty raw RFC 4648 standard base64 string (standard alphabet and canonical padding; no whitespace, URL-safe encoding, ordata:prefix), or a nonempty page-ownedBlob(includingFile).mediaTypeis required in both forms. A nonemptyBlob.typemust match it exactly. An empty Blob type is allowed whenmediaTypeis explicit. Validate decoded image format against the declared type; reject corrupt bytes and type mismatches withinvalid_request. A MIME label or file extension alone is insufficient validation.url, URI/file handles, extra metadata, audio, video, and PDFs areinvalid_request. Do not reuse chat's URL-resolution behavior for decisions. Neither the page transport nor a privileged host may fetch a URL found in decision input. The application may explicitly acquire a Blob before making a request, subject to its own permissions and CORS.
Implementations must document and enforce maximum attachment counts, decoded bytes per attachment, total decoded attachment bytes, and total encoded transport size (including base64 expansion and envelope overhead), in addition to the existing state/question limits. Count each shared attachment once per request, regardless of question count. Document any decoded image dimensions, pixel/frame limits, or other constraints used to bound decoding and native provider work. IPA sets no universal media byte/count ceiling. Avoid allocating unbounded decoded data: check encoded length or Blob size before reading or decoding, then validate actual bytes. Media base64 is not ordinary state text; measure UTF-8 state/questions and decoded/transport media budgets explicitly.
Validate schema, encoding, format, counts, and all locally known size limits
before provider inference I/O, and before prompting whenever possible. Once a
route is selected, check its MIME/format support and smaller limits, including
dimensions or animation constraints. Approval choices must exclude routes
known to be incompatible. A saved route is never silently replaced. A valid,
advertised image type unsupported on that route is unavailable; exceeding a
route limit is invalid_request. Known preflight failures yield no accepted.
Translate upstream size/count/format-limit rejections that could not be checked
locally to invalid_request, identifying the failing attachment index or budget
and the applicable limit/value when known. Yield no done on any failure.
The first-iteration snapshot includes array order, attachment objects, MIME
types, and exact bytes. Copy references to immutable Blobs or encode them before
an isolated-world hop; mutations or replacement of page arrays/objects during
encoding or approval must not change the snapshot. Validate and approve this
same snapshot and forward its exact bytes. No image bytes, state, questions,
or provider uploads may reach a provider before consent. Do not eagerly read
Blobs or start transport/provider work when request() merely returns an iterable.
Adapters must forward images as native media together with state to the native
decision API. Internal inline-data encodings and temporary provider uploads or
file handles are adapter details, never page-facing fields. Do not silently
omit, resize, truncate, replace images with OCR/captions/transcripts, emulate
decisions with chat, or split the request. If the route cannot consume the
complete native input, fail as above. The answers, per-question refusals, and
accepted → done stream are unchanged.
Cancellation must interrupt local encoding/validation, pending permission, uploads, and inference. Stop scheduling work and suppress late results even when underlying work cannot be interrupted. Iterator closure and navigation also cancel unfinished work. The transport owns releasing encoded copies, buffers, object URLs, and request references on success, denial, failure, abort, or closure. Adapters own cancelling uploads/inference and deleting temporary provider media on every exit where the provider supports deletion; otherwise document retention/expiry and do not reuse handles across requests or origins. No provider handles, credentials, or temporary-media URLs may leak to the page.
A successful stream is exactly accepted → done. There are no text,
reasoning, audio, or partial-answer deltas. done contains a non-whitespace
model, answers, and optional usage; it has no chat message, transcript,
or synthesis audio payload. usage uses inputTokens / outputTokens when
reported, each a nonnegative safe integer; omit unreported usage, never invent it.
answers must contain exactly one own entry for every requested id and no
extra ids. Each answer's type must match its question, except that any question
may instead return { type: "refusal" }. A refusal is an explicit provider
refusal for that question, not a missing answer, numeric zero, low confidence,
or permission denial. It carries no probability/choice/score fields. Other
questions can still succeed, and even an all-refusal result is a completed
request. Applications must check answer.type before reading answer fields.
For non-refused answers:
- Every probability and
confidencemust be finite and in[0, 1]. Noul has no separate confidence or distribution: near 0.5 means uncertainty about yes/no, rather than medium intensity. - Choice
probabilitiesmust have exactly the requested option keys.choicemust be one of the highest-probability keys; any tied maximum is valid. - Score
legendandprobabilitiesmust have exactly the zero-based decimal string indices"0"throughString(criteria.length - 1). Each legend value must equal the corresponding requested description.scoreis a finite probability-weighted level index in[0, criteria.length - 1], possibly fractional. Its difference fromsum(index * probability) / sum(probability)must be at most 0.02 to allow rounding. - Choice and score distributions must sum to 1 within an absolute tolerance
of 0.02. Preserve provider probabilities; do not silently renormalize a
malformed distribution.
confidencesummarizes concentration in that distribution; its formula is provider-defined in this draft. It is not a guarantee of correctness or a permission to act, and thresholds need evaluation for the application's data and selected model.
Implementations must validate the complete upstream result before yielding
done. Missing, extra, wrong-type, out-of-range, or inconsistent answers are
provider_error; do not expose a partial successful result or turn malformed
answers into refusals. Native wire shapes may be mapped (for example a boolean
predicate to noul, or ordered levels to zero-based score indices), but do not
emulate typed decisions by prompting a chat model and parsing generated text or
inventing probabilities. If the selected route cannot honor a well-formed
request, fail with unavailable and yield no done, except for applicable
resource-limit violations, which use invalid_request as specified above.
The normal cancellation rules apply during permission and provider work.
Aborting throws aborted, cancels in-flight provider work, and yields no
done. Stopping iteration must release resources and cancel unfinished work.
Decisions require a separate origin-and-method grant. A chat or speech grant
does not authorize decide; a decision grant does not authorize those methods.
A persistent grant binds the approved decision provider and model. Changing
that route requires fresh approval; changing global defaults must not rewrite
saved grants. Blocking an origin continues to block every method.
Decision media is a separate grant identity within decide, scoped to the
approved image MIME-type set as well as origin/provider/model. Existing or
migrated decision grants are text/JSON-only. Neither a text-only decision grant
nor a chat imageInput grant authorizes any decision attachment. A media request
requires a new prompt unless a decision-media grant covers every attachment
type; widening that type set requires fresh approval. A covering persistent
grant may authorize later images of those types within the implementation and
route limits without another prompt. It does not authorize URLs, other media
modalities, or a different route. Text-only requests may reuse the same route
without acquiring media permission.
Approval must identify the decision operation, provider/model, and question ids/types. For media requests it must also explicitly identify native image input and list every attachment by index, MIME type, and decoded byte size, plus the attachment count and total decoded bytes. Thumbnails are optional; if shown they must come from the approved snapshot without remote fetching. A state or question-content preview is optional and may be truncated, but the selected provider receives the complete bounded request. No request content may reach a provider before permission resolves. Result values never authorize the implementation to execute application actions; application code owns how to use the answers.
if (!window.inference?.getFeatures?.().methods?.decide) {
throw new Error("This IPA implementation does not support typed decisions.");
}
const controller = new AbortController();
try {
for await (const chunk of window.inference.request({
method: "decide",
state: { ticket: "My checkout page is blank after Pay." },
questions: {
is_bug: { type: "noul", instructions: "Is this a software defect?" },
team: {
type: "choice",
instructions: "Which team should handle this ticket?",
criteria: { billing: "Payments and refunds", technical: "Bugs and integrations", other: null },
},
urgency: {
type: "score",
instructions: "How urgent is the customer's problem?",
criteria: ["Routine", "Should be fixed soon", "Blocking revenue right now"],
},
},
signal: controller.signal,
})) {
if (chunk.type === "accepted") console.log("accepted");
if (chunk.type === "done") {
for (const [id, answer] of Object.entries(chunk.answers)) {
if (answer.type === "refusal") console.log(id, "refused");
else console.log(id, answer);
}
}
}
} catch (error) {
console.log(error.code); // permission_denied, invalid_request, unavailable, provider_error, aborted
}
// From a cancel control while awaiting permission/provider: controller.abort().The shared decision conformance assertions check successful stream order and answer invariants. Mock fixtures verify the contract; they do not certify live provider availability or model accuracy. The image inspection demo and its browser request module show separate MIME feature detection, local file input, refusal handling, cancellation, and unsupported selected routes. Media conformance support covers malformed/over-limit requests and approval snapshot/cancellation hooks.
Transcription and synthesis are optional variants of the same request function.
They use transcription and speech endpoints with operation-specific models and
stream shapes. They are distinct from chat audio: transcription promises a
faithful transcript, while chat may summarize or answer questions. STT and TTS
are capability categories, not additional method names.
transcribe accepts a prerecorded, finite media file containing a supported,
decodable audio track. Audio-only files and provider-supported video containers
are valid. The media object must contain exactly one source, checked by
property presence rather than truthiness:
{ data: string, mediaType }: non-empty raw base64, never a data URL. An explicit advertised media type is required; malformed base64 is rejected.{ data: Blob, mediaType? }: infer the media type fromBlob.typewhen omitted. If declared and detected types are present, their normalized values must agree.{ url: string, mediaType? }: fetch a non-empty URL in the page realm, under normal page CORS, and convert the response to a bounded source. Use a usable responseContent-Type; a declaration may supply a missing type, but a conflicting declared/detected type isinvalid_request. URL/network/CORS failures areinvalid_request; cancellation isaborted.
For example, { data: undefined, url: "..." } violates source exclusivity. A
privileged implementation must never fetch an arbitrary page-supplied media URL.
Page-side URL resolution can occur before approval, once iteration starts; no
media may be sent to a provider before approval.
Normalize MIME parameters, casing, and unambiguous aliases before comparison
(for example audio/mp3 to audio/mpeg, audio/x-m4a to audio/mp4, and
audio/wave, audio/x-wav, or audio/vnd.wave to audio/wav). Transcription
formats are capability-driven, not a universal allowlist. Advertise concrete
MIME types such as audio/mpeg, audio/mp4, audio/wav, audio/webm,
video/mp4, and video/webm only when supported. Provider adapters must declare
accepted types and should declare narrower container/audio-codec constraints
when reliably documented. Do not claim generic video/* support. Detect and
check narrower constraints when advertised; MIME/container acceptance alone
does not guarantee every audio codec works.
Malformed media, invalid base64, unsupported MIME/container/codec combinations
outside the advertised surface, and inputs exceeding a documented implementation
limit are invalid_request. A format mismatch specific to the selected route
after a well-formed advertised request is unavailable. Fail before accepted
whenever metadata makes the failure knowable; provider decoding can fail later.
A file without a supported audio track must fail with invalid_request and a
clear no-audio-track message. Provider-detected silence/no transcribable speech
must fail with a clear invalid_request, never an unexplained empty transcript.
Malformed provider responses or inconsistent stream results are provider_error.
language, when supplied, must be a structurally valid BCP 47 tag. Omission
means detect; it does not request translation. If the selected route cannot
faithfully represent a valid hint, fail with unavailable rather than discard it.
Only the audio track has transcription semantics. Visuals, subtitles, and other non-audio tracks must not influence the transcript; do not inspect frames or substitute multimodal chat/video understanding. Implementations may pass a provider-supported container through unchanged and are not required to demux, decode, extract, or transcode locally. If the selected route cannot accept the file as supplied, fail clearly instead of silently converting or dropping tracks. Ignoring visuals is a semantic guarantee, not a guarantee that visual bytes stay local: disclose when the complete container may leave the page.
synthesize.text must be a string containing non-whitespace text. Implementations
may set a documented maximum, measured in Unicode code points. The page supplies
no provider, model, voice ID, or voice sample. Those choices belong to the user
and implementation. Speech requests accept only their method-specific fields;
chat fields, page-selected route/voice fields, and unknown speech request or
media/output fields are invalid_request in this draft.
Omitting output or output.mediaType means audio/mpeg, including output: {}.
MP3 remains the default; WAV requires explicit output: { mediaType: "audio/wav" }
and an object advertisement containing audio/wav. Legacy
methods.synthesize: true means MP3 only. An object advertisement declares
exactly its nonempty supported format list; it does not implicitly include MP3.
Only the exact canonical values in SynthesisMediaType are valid output types;
unknown, aliased, or unadvertised types are invalid_request.
A well-formed advertised format unavailable on the user-selected or remembered
provider/model/voice route is unavailable. In particular, omitted output still
requests MP3 and fails with unavailable on a WAV-only route when MP3 is
advertised elsewhere; if MP3 is not advertised at all, it is invalid_request.
Filter approval choices by the requested format. Fail before accepted when
support is knowable, and never silently change encoding or providers. A saved
grant does not authorize switching routes to satisfy a different format.
Implementations must document conservative input/output limits and reject before
approval when source metadata makes that possible. The standard fixes no
provider-derived byte cap. Never truncate text or audio to fit a limit silently.
The common lazy iteration, permission, error, and cancellation rules apply.
Permission and preflight failures precede accepted. A successful stream yields
exactly one accepted, then method-specific chunks, then exactly one done;
nothing follows done. Media bytes may remain in the page through approval;
metadata needed for approval may cross earlier. Implementations may begin a
flow-controlled page-to-implementation upload after accepted. A failure during
transfer or provider work throws an InferenceError and yields no done.
Aborting before or after accepted, returning from the iterator, navigation,
and page closure must cancel active transfer/provider work.
Transcription yields zero or more append-only delta chunks, then
done.transcript.text. When deltas are emitted, their concatenation must equal
the final text exactly. Providers without streaming may return only the final
done after accepted; this is the explicit exception to delta concatenation.
Do not expose replacement/interim corrections, timestamps, or segment objects.
done.transcript.language is optional provider-reported language information.
usage.inputSeconds is optional input duration in seconds.
Synthesis yields one or more audio_delta chunks, even for a non-streaming
upstream response. Every chunk contains Uint8Array encoded bytes and
the requested media type (including the MP3 default). Every audio_delta.mediaType
and done.audio.mediaType must match that type. Concatenating bytes in order
must reproduce exactly one complete playable file, with the original encoded
bytes preserved. done.audio.byteLength must equal that byte count; done.audio is
metadata and must not duplicate bytes. usage.inputCharacters, when reported,
counts input Unicode code points; usage.outputSeconds optionally reports output
duration. Usage is optional and must not be fabricated. Counts and durations
must be finite and nonnegative; byte lengths must be positive safe integers.
For WAV, chunks are transport fragments of one RIFF/WAVE file, not independent
WAV files. The complete file has a single RIFF container with valid sizes,
format information, and audio data; do not prepend a header to every chunk or
label headerless PCM as WAV. A complete unary provider result may be split into
bounded byte chunks with the same backpressure and cancellation rules. Invalid
containers, mismatched MIME types, truncated output, or inconsistent byte counts
are provider_error and must not yield a successful done.
IPA returns bytes and never starts playback. Pages may construct a Blob using
the returned media type, attach controls, and play only following an explicit user
action, subject to browser autoplay policy. See the browser-only
speech examples.
Existing grants are chat-only. Transcription and synthesis require separate origin-and-method grants. A persistent grant binds the exact provider/model and, for synthesis, voice. Changing that route requires fresh approval; changing global defaults must not silently rewrite a saved grant. Blocking an origin continues to block every method.
Transcription approval must identify the operation, provider/model, media type, byte size, and container/audio codec/duration when known. For a video container, disclose that the complete file may be uploaded even though only audio is transcribed. Synthesis approval must identify generated speech and show the provider/model/voice and requested output format. A page cannot override the route choice.
Implementations must update privacy disclosures before shipping speech. Recordings
can contain biometric voice data, bystander speech, location/health information,
and background conversations. Extensions must not capture the microphone or
autoplay speech. A page may use browser microphone permission to create a
recording, stop recording, then submit the completed bounded Blob with separate
IPA transcription approval; a live microphone stream is out of scope.
The API is available only to top-level pages in a secure context that have a
tuple origin — typically https: and loopback http: (localhost,
127.0.0.1, [::1]). Opaque origins, including file: documents, are not
supported: permission is scoped to the page's origin, and those contexts do not
provide a stable site identity for grants or blocks.
A persistent allow grant records the provider and model chosen at approval time, scoped by origin and method (and voice for synthesis); later requests reuse that approved route. Existing grants migrate as chat-only and authorize neither decisions nor speech. Changing the approved route requires fresh approval. Blocking an origin blocks every method. Changing a global default provider or model does not rewrite existing origin grants. Implementations must validate requests and must not expose API keys or provider credentials to page scripts.
Implementations that call local inference servers (for example Ollama on
loopback) should avoid requiring users to widen the server's allowed origins.
Browser extension requests often include a chrome-extension:// (or similar)
Origin header that many local servers reject with HTTP 403. Prefer stripping
or rewriting that header on extension-initiated requests to the local server so
default local installs work without configuring OLLAMA_ORIGINS or equivalent
allowlists. Broad wildcards such as chrome-extension://* remain an optional
fallback, not the recommended default.
Implementations may show a truncated preview of request content in the permission UI to support informed consent. Previewing content is optional; applications must not assume the UI reveals message content, including role: "tool" payloads on a later round of the same permission episode. Implementations may emphasize some messages and collapse others (for example by role) without changing the request sent to the provider.
Function tools run in the page. A site that offers tools can cause the model to request those functions with attacker-influenced arguments (including via retrieved or user-supplied text). Implementations must not execute page-defined tools, and must not treat a chat-only persistent grant as consent for tools. Listing function names at approval time is required so the user can refuse a tools request independently of chat.
Hosted web search is implementation- or provider-executed, not page-executed. The page never sees web_search toolCalls and must not run a page-side search handler. A chat-only grant and a function-tools grant each do not cover { type: "web_search" }. Listing a hosted-search label at approval time is required so the user can refuse public-web lookup independently of chat and of page function tools. Implementations that fetch result pages (not only a search index) must disclose that fetch, not only “web search”.
Image parts in messages and output.images are each distinct from chat, function tools, and hosted search. A chat-only grant covers neither. Listing an image-input label when messages contain ImageParts, and an image-output label when output.images is true, is required so the user can refuse photos leaving the page independently of receiving generated media. Thumbnails in the permission UI are optional. The host must not fetch { url } itself; page-side resolve keeps local models offline.
Decision attachments have their own media grant and approval. Chat image grants and existing text/JSON decision grants do not cover them. Native image bytes are part of the immutable approved input; neither ordinary URLs nor media-shaped values in decision state cause network fetching.
method: "image"(would duplicate chat and force a later migration)- Image parts on
systemortoolmessages - Streaming partial images (
image_delta) - Size / quality / aspect-ratio enums (later
optionskeys, ignore-if-unsupported) - Embeddings
- Decision audio/video/PDF attachments, remote media URLs, provider file handles, per-question attachments, structured instructions/criteria, additional question types, partial answers, and page-selected decision routes or generation controls
- Live/unbounded audio, realtime or duplex speech sessions, and conversational audio understanding through chat
- Transcription translation, timestamps, segments, confidence, diarization, and transcript replacement
- Synthesis formats other than MP3 and WAV (including raw PCM), page-selected model/voice IDs, voice cloning, custom voice enrollment, multi-speaker audio, and speaking-rate/style controls
- Video understanding, frame inspection, OCR, or subtitle extraction through transcription
- Image generation as a page-facing
{ type: "image_generation" }tool (content parts +output.imagesis the canonical surface) - Other hosted / provider-executed tools (for example MCP)
- Options on
{ type: "web_search" }such as search filters, recency, or location - Structured citations / a
sourcesfield ondone(links may appear as ordinary text inmessage.content) - Streaming
toolCalldeltas - An in-API agent loop (the page still owns the function-tool loop; hosted search turns stay hidden)
Estimated cost in the permission UI is optional extension UX and is not required by this draft.