Decision models

@e2e-dev/decision runs agent.act and agent.assert through a decision model instead of an LLM agent. For each action, the decision model picks an operation, then an element, from the goal, the recent actions, and the newest semantic tree. It never writes free text: a small text model writes field values. The runner still authorizes every action.

Any AI SDK decision model that answers choice questions with probabilities works, such as TypeSafe Jev or OpenAI gpt-6-luna. evaluationModel() instances work too. A model id string or a language model in model fails the config load with INVALID_CONFIG.

The same package’s decisionFilter() runs only the tests a pull request can affect. See Test filters.

#Install and configure

Install the package, a decision model provider, and a text model provider. Decision models need ai 7.0.134 or later.

npm install -D @e2e-dev/decision @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider

pnpm add -D @e2e-dev/decision @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider

bun add -d @e2e-dev/decision @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider

Set the executor on the agents entry:

import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { decisionExecutor } from '@e2e-dev/decision';
import { typeSafeAi } from '@ai-sdk/typesafe-ai';
import { openrouter } from '@openrouter/ai-sdk-provider';

export default {
  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
  agents: {
    default: {
      executor: decisionExecutor({
        model: typeSafeAi.decisionModel('jev-latest'),
        textModel: openrouter('inception/mercury-2.5'),
      }),
    },
  },
} satisfies E2EConfig;

Tests are plain agent.act and agent.assert calls:

import { test, expect } from 'e2e';

test('adds a todo', async ({ app, screen, agent }) => {
  await app.open('/todos');
  await agent.act('Add a todo named Buy coffee.');
  await expect(screen.getByTestId('todo')).toContainText('Buy coffee');
  await agent.assert('The todo list contains Buy coffee.');
});

npm install -D @ai-sdk/openai

pnpm add -D @ai-sdk/openai

bun add -d @ai-sdk/openai

import { openai } from '@ai-sdk/openai';

const executor = decisionExecutor({
  model: openai.decisionModel('gpt-6-luna'),
  textModel: openai('gpt-6-luna-fast'),
  vision: true,
});

#What the model can do

Each turn offers the operations the engine supports on the elements of the newest screen: tap, type, typeSecret, submit, select, check, upload, hover, secondary_tap (a right-click), double_tap, long_press, drag, scroll_to, scroll_up, scroll_down, and back. Hover, right-click, drag, and scroll-to also reach labeled nodes that are not controls, such as a card that opens a menu on hover, and a drag can drop onto any of them. tap also reaches text fields, to open a date picker or a suggestion list. While an on-screen keyboard shows on an engine that can dismiss it, such as the mobile engine, the executor also offers dismiss_keyboard, even when the keyboard is missing from the tree.

Every operation that acts on an element asks for its target, even when only one element fits it. Every target question has a none option, so a provider with no fitting target answers none instead of refusing. none blocks the step.

#How text gets typed

When the decision model picks type, the text model writes the value from the goal, the params, the field’s label, and the page text. The executor types exactly that string. With no value in the goal, it types nothing and the decision model picks again. For upload, the text model lists the project-relative paths the goal or params name, and the runner authorizes each one. Model output never becomes a selector, a URL, or a key.

Password fields never reach the text model. Only typeSecret fills a secret, from its declared handle.

The text model also answers waitFor and extract. Instead of textModel, you can set model on the agents entry. With neither, the executor never offers type, upload, tap_at, or type_at.

#Finishing a step

When the decision model answers done or failed, the executor checks the claim on a fresh screen with one question: is the step’s task complete? The check never sees model reasoning or a secret.

A confirmed done passes the step, and a confirmed failed fails it with ACTION_FAILED. An unconfirmed claim goes back to the decision model as history. A second one ends the step: ACTION_FAILED when the screen contradicts the goal, else blocked with AUTOMATION_UNSUPPORTED.

#Gates

Both gates are off by default, so the executor follows the most probable choice. Set minProbability or minConfidence to block weaker actions, reject weaker completion checks, and fail weaker assertions as inconclusive:

const executor = decisionExecutor({ model, minProbability: 0.9, minConfidence: 0.8 });

Probability is the chosen option’s share of the answer distribution. Confidence is the provider’s own statistic, 0 when it reports none. OpenAI rounds both to two decimals. Measure accuracy on your own tests before you set a threshold.

#Vision

vision: true adds a masked screenshot to every observation. Assertions and completion checks see it, so the model can judge drawn content the tree doesn’t show. The operation and target questions read the tree only. agent.assert(..., { vision: true }) and vision: 'only' work too.

The model must accept images and answer score questions. TypeSafe Jev does with @ai-sdk/typesafe-ai 3.0.19 or later and a TypeSafe API key in TYPESAFE_AI_API_KEY:

import { typeSafeAi } from '@ai-sdk/typesafe-ai';
import { openrouter } from '@openrouter/ai-sdk-provider';

const executor = decisionExecutor({
  model: typeSafeAi.decisionModel('jev-latest'),
  textModel: openrouter('inception/mercury-2.5'),
  vision: true,
});

openai.decisionModel('gpt-6-luna') takes images too. Vercel AI Gateway rejects the screenshot for every decision model, so keep vision off there.

With a text model, vision also offers tap_at for a control the tree doesn’t list. The text model names the control, such as “the 3 key on the keypad”, and the decision model finds it on the screenshot, then on two zoomed crops. A point tap costs four decide calls and one text call, or one of each to repeat a tap on the same page.

When the engine can type into whatever has focus, vision also offers type_at for a drawn field. The text model names the field and the value from the goal, the decision model finds the field on the screenshot, and the executor taps it and types the value at the caret, after any text already there. It costs two decide calls and one text call.

#Provider options

The executor’s providerOptions go with every decide call, completion checks included, and reach the decision model only. Use them for options such as zero data retention on Vercel AI Gateway:

import { gateway } from 'ai';

const executor = decisionExecutor({
  model: gateway.decisionModel('typesafe-ai/jev'),
  textModel: gateway('openai/gpt-6-luna-fast'),
  providerOptions: { gateway: { zeroDataRetention: true } },
});

The text model takes the agents entry’s providerOptions, so a setting for one model, such as a gateway fallback list, never reaches the other.

AI Gateway can rerun an uncertain decision on another model. Add the conditional entry to gateway.models; it reaches every decide call, and a triggered fallback bills both models:

const executor = decisionExecutor({
  model: gateway.decisionModel('typesafe-ai/jev'),
  textModel: gateway('openai/gpt-6-luna-fast'),
  providerOptions: {
    gateway: { models: [{ model: 'openai/gpt-6-luna-decisions', when: { confidenceBelow: 0.6 } }] },
  },
});

A decision model fallback keeps the gates: the executor reads the fallback model’s probabilities and confidence. A language model fallback, such as openai/gpt-6-luna-fast, answers with a choice and no distribution. The executor takes that choice with probability 1 and confidence 0, so a minConfidence gate blocks every escalated answer.

#Limits

• No navigate. In-app navigation is a tap on a link. URLs belong in test code (browser.goto, app.open).
• No multi-select listbox and no scrollUntil. type_at never clears a drawn field first, so it suits empty fields.
• The text model that names a drawn control doesn’t see pixels. It can’t follow an instruction that exists only as drawn text, such as a wizard that paints “tap the circle”.
• Every decide call and text call counts against maxModelCalls.
• Three actions in a row that fail or change nothing block the step with AUTOMATION_UNSUPPORTED.
• A question the provider refuses fails the step with MODEL_OUTPUT_INVALID.
• A question has at most 255 options, TypeSafe’s limit. On a larger screen, nodes in the viewport come first and the rest count as omitted, so the model can scroll.

#Next