Skip to content

Commit 8db3629

Browse files
thymikeethymikee
authored
feat(web): add hover command for hover-gated UI (#1783) (#1786)
* feat(web): add hover command for hover-gated UI (#1783) Add a first-class `hover <x y|@ref|selector> [--settle]` verb, admitted on web only, that moves the pointer without pressing via the agent-browser backend (mouse move). It rides the existing targeted-touch pipeline (ref/selector/coordinate resolution, occlusion/off-screen guards, settle observation, response builder, recording) through a new optional Interactor/backend `hover` op that only the web provider implements. Touch platforms have no hover state: capabilities advertise it on web only and iOS/Android/Linux reject it at admission with a --platform web hint; longpress stays the mobile hold-gesture verb. Closes #1783 * fix(hover): native hoverRef route for web @ref, android coverage pin, revert skill edit Review follow-ups on #1786: - hover @ref on web now dispatches through the provider's own element handle (agent-browser `hover <ref>`) via a new backend hoverTarget, mirroring click/fill's ADR 0011 native-ref path — web ref frames carry no rects, so the coordinate route could never resolve them. The shared preflight + exact-ref dispatch is extracted into dispatchNativeRefInteraction and used by tap/fill/hover; the guarantee matrix native-ref row now lists hover. - Daemon regression test is production-faithful: rect-less web ref frame, scoped provider, asserts no coordinate dispatch. Selector→coordinate and provider hoverRef tests added. - Android emulator coverage summary pin 2/53 → 3/54. - skills/agent-device/SKILL.md reverted (out of scope, AGENTS.md rule). - Docs/help disclose that --settle with @ref on web shares click's existing limitation; use a selector or coordinates for the settled diff. * test: drive hover through the apple output guard; cover direct hover dispatch The provider-integration apple-leak guard partitions every public command into driven/skipped; hover was neither, which failed Integration Tests and took Coverage down with it. Drive it (it reaches the Apple capability refusal, which is scanned like any other error response). Also cover the direct-dispatch handleHoverCommand seam. * test(web): drive hover @ref in the provider-backed web scenario The integration-progress gate requires every public command to be referenced by a provider-backed scenario. Add hover @ref to the web desktop flow: it must reach the provider's hoverRef handle (never a coordinate) and be recorded on the session without fabricated x/y, like click @ref.
1 parent 8b0560a commit 8db3629

62 files changed

Lines changed: 777 additions & 94 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Unreleased
44

5+
- New `hover <x y|@ref|selector>` command for `--platform web` (#1783). It moves the pointer over the target without pressing, so hover-gated UI — a message row's `...` toolbar, a menu that opens on pointer enter — becomes reachable through agent-device the way it already was through the underlying `agent-browser` backend (`mouse move`). It is a member of the targeted-touch family: same `@ref`/selector/coordinate targeting, occlusion and off-screen guards, and `--settle` (the settled diff carries the revealed controls with fresh refs, e.g. `+ @e4 [button] "Delete"`), but no `--verify`, since hover reveals rather than activates. `hover @ref` publishes as a portable selector line in recorded scripts, and the Node client exposes `interactions.hover`. Hover is a pointer state that touch platforms do not have, so `capabilities` advertises it on web only and iOS/Android/Linux reject it during admission with `UNSUPPORTED_OPERATION` and a hint naming `--platform web`; `longpress` remains the mobile hold-gesture verb.
56
- Internal: session recording is now derived from the script-publication lifecycle instead of being stored beside it. `SessionState.recordSession` is removed; `isRecordingPublication` answers the question from the aggregate (ordinary authoring records only while ARMED, a repair transaction records for its whole lifetime). The stored flag was a second source of truth that handler surfaces set directly, which is how #1533's aborted-recording drift arose — no surface can now arm recording without moving the lifecycle that authorizes it, and the script writer's publication gate is answered entirely by the aggregate. Behavior-preserving: the derivation reproduces what the flag held at every transition.
67
- Fixed: a script recording aborted by a second `open` is no longer published by a later bare `close` (#1533). `open <app> --save-script` followed by a second successful `open` terminates the recording and warns "Script publication was aborted…", and `close --save-script` correctly refuses it with "Retry with plain close; it will tear down the session without writing." But when that second `open` itself carried `--save-script`, the flag re-armed recording behind the terminal status, and a bare `close` then wrote the full session log to disk — publishing a recording the caller had been told was aborted, and breaking the promise the refusal makes. An aborted authoring lifecycle is now terminal by construction: `--save-script` arms nothing on any surface that handles it — the re-open builder, the close finalizer, and the recorded-action ingress — and the script writer refuses the lifecycle from every path that reaches it (bare `close`, teardown, idle-reap, active publication). This also stops an aborted session from paying recording-time costs it can never publish: a re-opened aborted recording no longer keeps the direct iOS selector fast paths for `click` and `get` disabled. Armed recordings, published recordings, and every repair transaction are unaffected.
78
- `agent-device mcp` now serves the stateless MCP `2026-07-28` revision alongside the handshake-based revisions it already spoke, as the spec's "dual-era server". Modern clients probe `server/discover`, which advertises the supported revisions, the tools capability, and server identity; their requests declare a protocol version in `_meta`, and their results carry `resultType: "complete"` plus `_meta["io.modelcontextprotocol/serverInfo"]`. `tools/list` and `server/discover` now return the `ttlMs`/`cacheScope` cache hints, so a client can cache the 55-tool, ~223KB tool list for an hour instead of re-fetching it on every start; the list was already emitted in a deterministic (sorted) order, which is the other half of what makes it cacheable. Each revision is answered on its own wire contract: a request declaring `2025-11-25` or `2025-06-18` through modern framing still gets the legacy result shape, and `initialize` never agrees to `2026-07-28`, which has no handshake to establish. A declared revision this server does not implement is rejected with `UnsupportedProtocolVersionError` (`-32022`) naming the ones it does, rather than being served under a version the client did not ask for, and modern framing that omits its required `protocolVersion`/`clientCapabilities` metadata — or supplies a `clientInfo` that is not a valid `Implementation` — is rejected as invalid params. `initialize` and `ping` were removed in `2026-07-28`, so a modern-framed call to either is answered `-32601` rather than served inside a `resultType: "complete"` envelope. Responses to legacy clients are unchanged byte-for-byte — `initialize` and `ping` are still served, and no cache, `resultType`, or `_meta` field is added to their results. Nothing here affects the CLI, Node, or daemon surfaces: the stdio transport, the tool set, and every tool's input/output schema are untouched.

‎packages/ad-script/src/internal/script-utils.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,10 @@ export function isClickLikeCommand(command: string): command is 'click' | 'press
4444
return command === 'click' || command === 'press';
4545
}
4646

47-
export function isTouchTargetCommand(command: string): command is 'click' | 'press' | 'longpress' {
48-
return isClickLikeCommand(command) || command === 'longpress';
47+
export function isTouchTargetCommand(
48+
command: string,
49+
): command is 'click' | 'press' | 'longpress' | 'hover' {
50+
return isClickLikeCommand(command) || command === 'longpress' || command === 'hover';
4951
}
5052

5153
function isTypingCommand(command: string): command is 'type' | 'fill' {

‎packages/ad-script/src/internal/script.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -463,10 +463,11 @@ function parseReplayScriptLine(line: string): SessionAction | null {
463463
return action;
464464
}
465465

466-
// wait @ref [timeout] and longpress @ref [durationMs] flow through this
467-
// generic branch: strip recorded generation pins like the branches above.
466+
// wait @ref [timeout], longpress @ref [durationMs], and hover @ref flow
467+
// through this generic branch: strip recorded generation pins like the
468+
// branches above.
468469
action.positionals =
469-
command === 'wait' || command === 'longpress'
470+
command === 'wait' || command === 'longpress' || command === 'hover'
470471
? args.map((token) => stripRecordedRefGeneration(token))
471472
: args;
472473
return action;

‎packages/contracts/src/client-gesture.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,11 @@ export type LongPressOptions = DeviceCommandBaseOptions &
6464
durationMs?: number;
6565
};
6666

67+
export type HoverOptions = DeviceCommandBaseOptions &
68+
SelectorSnapshotCommandOptions &
69+
InteractionTarget &
70+
SettleCommandOptions;
71+
6772
export type SwipeOptions = DeviceCommandBaseOptions & {
6873
from: { x: number; y: number };
6974
to: { x: number; y: number };

‎packages/contracts/src/facades/client.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ export type {
4747
FillOptions,
4848
FlingOptions,
4949
FocusOptions,
50+
HoverOptions,
5051
LongPressOptions,
5152
PanOptions,
5253
PinchOptions,

‎packages/contracts/src/facades/interaction.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,8 @@ export type {
8585
FillCommandResponseData,
8686
FillCommandResult,
8787
FindCommandResponseData,
88+
HoverCommandResponseData,
89+
HoverCommandResult,
8890
InteractionEvidence,
8991
InteractionTarget,
9092
LongPressCommandResponseData,

‎packages/contracts/src/interaction-guarantees.ts‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ const RUNTIME_TREE_SHARED_GUARANTEES = {
199199
export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPathContract> = {
200200
'runtime-selector': {
201201
description: 'Daemon tree capture, selector chain resolution, guarded coordinate tap.',
202-
commands: ['press', 'click', 'fill', 'longpress'],
202+
commands: ['press', 'click', 'fill', 'longpress', 'hover'],
203203
guarantees: {
204204
...RUNTIME_TREE_SHARED_GUARANTEES,
205205
disambiguation: {
@@ -221,7 +221,7 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
221221
'runtime-ref': {
222222
description:
223223
'Session snapshot ref lookup, guarded coordinate tap. #1654: when the caller already resolved the node (a mutating `find`), the lookup is replaced by that node and every guarantee below is enforced against it — the guards are unchanged, only the lookup is skipped.',
224-
commands: ['press', 'click', 'fill', 'longpress'],
224+
commands: ['press', 'click', 'fill', 'longpress', 'hover'],
225225
guarantees: {
226226
...RUNTIME_TREE_SHARED_GUARANTEES,
227227
disambiguation: {
@@ -361,13 +361,13 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
361361
},
362362
'native-ref': {
363363
// WEB-ONLY in production: apple/android backends never define
364-
// tapTarget/fillTarget - the sole wiring is the web provider's clickRef
365-
// (a stable DOM-handle click). Verified 2026-07-04 while designing the
366-
// #1088 retirement experiment, which this finding dissolved: there is no
367-
// iOS runner round trip to retire.
364+
// tapTarget/fillTarget/hoverTarget - the sole wiring is the web provider's
365+
// clickRef/fillRef/hoverRef (stable DOM-handle actions). Verified
366+
// 2026-07-04 while designing the #1088 retirement experiment, which this
367+
// finding dissolved: there is no iOS runner round trip to retire.
368368
description:
369-
'click @ref / fill @ref dispatch to backend.tapTarget/fillTarget (web provider clickRef only; no mobile backend implements these) without runtime resolution when no non-default options are set. A zero-round-trip preflight (preflightNativeRefInteraction) runs the shared guards against the stored session snapshot node first; no snapshot / no usable rect makes the preflight a no-op.',
370-
commands: ['click', 'fill'],
369+
'click @ref / fill @ref / hover @ref dispatch to backend.tapTarget/fillTarget/hoverTarget (web provider clickRef/fillRef/hoverRef only; no mobile backend implements these) without runtime resolution when no non-default options are set. A zero-round-trip preflight (preflightNativeRefInteraction) runs the shared guards against the stored session snapshot node first; no snapshot / no usable rect makes the preflight a no-op.',
370+
commands: ['click', 'fill', 'hover'],
371371
guarantees: {
372372
disambiguation: {
373373
kind: 'inapplicable',
@@ -428,7 +428,7 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
428428
},
429429
coordinate: {
430430
description: 'Raw x/y tap. Semantics are intentionally minimal.',
431-
commands: ['press', 'click', 'fill', 'longpress'],
431+
commands: ['press', 'click', 'fill', 'longpress', 'hover'],
432432
guarantees: {
433433
disambiguation: {
434434
kind: 'inapplicable',

‎packages/contracts/src/interaction.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,15 @@ export type LongPressCommandResponseData =
371371
| (TouchResponseRef & TouchLongPressExtras)
372372
| (TouchResponseSelector & TouchLongPressExtras);
373373

374+
type TouchHoverExtras = {
375+
gesture: 'hover';
376+
};
377+
378+
export type HoverCommandResponseData =
379+
| (TouchResponsePoint & TouchHoverExtras)
380+
| (TouchResponseRef & TouchHoverExtras)
381+
| (TouchResponseSelector & TouchHoverExtras);
382+
374383
/**
375384
* Internal runtime result for press/click. The daemon response layer turns
376385
* this into `PressCommandResponseData` via `buildInteractionResponseData`.
@@ -408,6 +417,17 @@ export type LongPressCommandResult = ResolvedInteractionTarget & {
408417
settle?: SettleObservation;
409418
};
410419

420+
/**
421+
* Internal runtime result for hover. The daemon response layer turns this
422+
* into `HoverCommandResponseData` via `buildInteractionResponseData`.
423+
*/
424+
export type HoverCommandResult = ResolvedInteractionTarget & {
425+
backendResult?: Record<string, unknown>;
426+
message?: string;
427+
warning?: string;
428+
settle?: SettleObservation;
429+
};
430+
411431
/**
412432
* Daemon response data for the `find` command. Read-only actions (`exists`,
413433
* `wait`, `get_text`, `get_attrs`) may issue a pinnable ref with

‎packages/contracts/src/interactor-types.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,12 @@ export type Interactor = {
195195
tapElementSelector?(selector: ElementSelectorTapOptions): Promise<Record<string, unknown> | void>;
196196
doubleTap(x: number, y: number): Promise<Record<string, unknown> | void>;
197197
longPress(x: number, y: number, durationMs?: number): Promise<Record<string, unknown> | void>;
198+
/**
199+
* Move the pointer to a point without pressing. Only pointer-driven
200+
* platforms (web today) implement it; touch platforms have no hover state
201+
* and leave it undefined, which the `hover` command reports as unsupported.
202+
*/
203+
hover?(x: number, y: number): Promise<Record<string, unknown> | void>;
198204
focus(x: number, y: number): Promise<Record<string, unknown> | void>;
199205
type(text: string, delayMs?: number): Promise<TypeTextBackendResult | void>;
200206
fillElementSelector?(

‎src/__tests__/test-utils/property-arbitraries.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,7 @@ const REPLAY_SCRIPT_LINE_PLANS = {
357357
.map(([target, button]) => `press ${target} --button ${button}`),
358358
),
359359
longpress: scriptTargetArb.map((target) => `longpress ${target}`),
360+
hover: scriptTargetArb.map((target) => `hover ${target}`),
360361
wait: fc
361362
.tuple(scriptTargetArb, fc.integer({ min: 100, max: 5000 }))
362363
.map(([target, timeout]) => `wait ${target} ${timeout}`),

0 commit comments

Comments
 (0)