Skip to content

Commit a0ab735

Browse files
authored
fix(interaction): direction-named, selector-first off-screen recovery hints (#1366) (#1374)
* fix(interaction): direction-named, selector-first off-screen recovery hints (#1366) An agent that hits a scrollable form section gets correctly rejected for targeting an off-screen element, but the recovery hint didn't name a concrete next move. Its "obvious" retry — re-issue the same @ref after scrolling — is exactly what both live guards reject: the off-screen guard still sees it off-screen, and the scroll expired the ref frame (ADR 0014), so the @ref is refused as stale. Two bootstrap-bench runs burned all 60 turns on a single checkout-form screen this way (arm-independent). Make the rejection self-sufficient for recovery instead of relaxing either guard: - Off-screen selector/ref rejection now names the exact scroll direction (computed from the target rect vs its effective viewport) and steers the retry to a *selector*, which re-resolves against a fresh snapshot and bypasses the ref-frame admission guard entirely. The direction is also surfaced as a machine-readable `scrollDirection` detail. - `scroll`'s unknown-direction error now carries a grammar hint, since the transcripts show agents mis-shaping it as `scroll @ref down` — scroll takes a direction and no target. New geometry helper `classifyOffscreenScrollDirection` maps a rect + viewport to the reveal direction (largest-overshoot axis), following the same convention the CLI off-screen summary already uses. Both guards keep their exact rejection semantics and messages; only the hints and one added detail field change. * fix(interaction): derive scroll direction from the boundary that rejected (#1366 review) Address P1: the direction classifier used a rect-vs-single-viewport test, so it only fired when the whole rect was fully past an effective-viewport edge. But `isNodeVisibleOnScreen` rejects on two boundaries — and the second one (tap-point CENTER outside the ROOT viewport while the rect still overlaps its container) is exactly the off-screen-drawer / edge-straddling case #1366 is about. Those got the generic hint and no scrollDirection, leaving the loop unresolved. `classifyOffscreenScrollDirection` now takes (node, nodes) and mirrors both rejection boundaries: full separation from the effective viewport, then a center pushed outside the root viewport. Direction is taken from whichever boundary failed, using the same unrounded center as the tap-point rule, so a rejected target always yields a direction. Regressions: partial clip whose center is past the viewport edge (both the pure classifier and the live selector-press rejection path), and a child inside an off-screen scrollable ancestor (closed drawer). * style: oxfmt the windowRoot test helper (#1366 CI) * fix(interaction): bounded off-screen recovery hint — a large scroll overshoots (#1366 review) Live evidence exposed that the hint over-promised: it advertised `scroll <direction>` -> retry-selector as the terminating recovery, but on the motivating checkout form a plain `scroll` (iOS fling momentum) repeatedly overshoots the narrow pressable band and the loop never terminates — even bounded `scroll --pixels 200` oscillates up/down. A momentum-free `gesture pan` in small steps lands the target (verified: 3–4 attempts -> Tapped "Cash"). Revise the hint to prescribe BOUNDED movement in the named direction, retrying the same selector after each step, and to name the overshoot failure mode and the reliable `gesture pan` fallback. Direction + selector-first steering are unchanged. `scrollRevealClause` drops the "to bring it on-screen" over-promise. Tests assert the bounded-recovery guidance (small steps / gesture pan) on both a selector and a ref off-screen path so it can't silently regress.
1 parent d451906 commit a0ab735

8 files changed

Lines changed: 320 additions & 10 deletions

File tree

‎src/commands/interaction/interactions.test.ts‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
import { expect, test } from 'vitest';
2-
import { interactionDaemonWriters } from './interactions.ts';
2+
import { interactionCliReaders, interactionDaemonWriters } from './interactions.ts';
3+
import { AppError } from '../../kernel/errors.ts';
4+
import type { CliFlags } from '../cli-grammar/flag-types.ts';
5+
6+
const BASE_FLAGS: CliFlags = { json: false, help: false, version: false };
37

48
test('swipe writes only typed daemon input', () => {
59
const request = interactionDaemonWriters.swipe({
@@ -19,3 +23,18 @@ test('swipe writes only typed daemon input', () => {
1923
pattern: 'ping-pong',
2024
});
2125
});
26+
27+
test('scroll reader rejects a @ref in the direction slot with a grammar hint (#1366)', () => {
28+
try {
29+
interactionCliReaders.scroll(['@e29'], BASE_FLAGS);
30+
expect.unreachable('scroll should reject a ref where a direction is expected');
31+
} catch (error) {
32+
expect(error).toBeInstanceOf(AppError);
33+
const appError = error as AppError;
34+
expect(appError.code).toBe('INVALID_ARGS');
35+
// The hint teaches the direction-first grammar and that scroll takes no target,
36+
// so the agent stops cycling `scroll @ref down` / `scroll down @ref`.
37+
expect(appError.details?.hint).toMatch(/direction first/i);
38+
expect(appError.details?.hint).toMatch(/no @ref or selector/i);
39+
}
40+
});

‎src/commands/interaction/interactions.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,12 @@ function readScrollDirection(value: string | undefined): ScrollInputDirection {
177177
) {
178178
return value;
179179
}
180-
throw new AppError('INVALID_ARGS', `Unknown direction: ${String(value)}`);
180+
// #1366: agents recovering from an off-screen target often mis-shape this as
181+
// `scroll @ref down` or `scroll down @ref`. scroll takes no target — name the
182+
// grammar so the retry lands instead of cycling through arg orders.
183+
throw new AppError('INVALID_ARGS', `Unknown direction: ${String(value)}`, {
184+
hint: 'scroll takes a direction first, then an optional amount: scroll <up|down|left|right|top|bottom> [amount]. It takes no @ref or selector — scroll the whole viewport, then retry the target with a selector.',
185+
});
181186
}
182187

183188
function readLongPressTargetPositionals(positionals: string[]): {

‎src/commands/interaction/runtime/interactions.test.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,10 @@ test('native ref click preflight refuses an off-screen ref without calling the b
160160
const details = (error as { details?: Record<string, unknown> }).details;
161161
assert.equal(details?.reason, 'offscreen_ref');
162162
assert.equal(details?.ref, 'e2');
163-
assert.ok(typeof details?.hint === 'string');
163+
// #1366: closed drawer sits left of the viewport -> `scroll left`, selector-first.
164+
assert.equal(details?.scrollDirection, 'left');
165+
assert.match(String(details?.hint), /scroll left/i);
166+
assert.match(String(details?.hint), /selector/i);
164167
return true;
165168
},
166169
);

‎src/commands/interaction/runtime/resolution.test.ts‎

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,58 @@ test('runtime press refuses a selector that resolves to an off-screen element',
120120
assert.match(error.message, /off-screen element and is not safe to press/);
121121
const details = (error as { details?: Record<string, unknown> }).details;
122122
assert.equal(details?.reason, 'offscreen_selector');
123-
assert.ok(typeof details?.hint === 'string');
123+
// #1366: the closed-drawer shape sits fully left of the viewport, so the
124+
// hint names `scroll left` and steers back through the same selector.
125+
assert.equal(details?.scrollDirection, 'left');
126+
assert.match(String(details?.hint), /scroll left/i);
127+
assert.match(String(details?.hint), /selector/i);
128+
return true;
129+
},
130+
);
131+
assert.equal(taps.length, 0);
132+
});
133+
134+
test('runtime press names a direction for a partial clip whose center is off-screen', async () => {
135+
// #1366 regression: the row still OVERLAPS the viewport (top edge inside), so
136+
// the rect-vs-viewport form yields no direction — but its tap-point center is
137+
// below the bottom edge, which is what the visibility guard rejects. The hint
138+
// must still name `scroll down` rather than falling back to the generic phrasing.
139+
const partialClipSnapshot = makeSnapshotState([
140+
{
141+
index: 0,
142+
depth: 0,
143+
type: 'Application',
144+
rect: { x: 0, y: 0, width: 400, height: 800 },
145+
hittable: true,
146+
},
147+
{
148+
index: 1,
149+
depth: 2,
150+
parentIndex: 0,
151+
type: 'Button',
152+
label: 'Cash',
153+
rect: { x: 20, y: 790, width: 200, height: 44 },
154+
hittable: true,
155+
},
156+
]);
157+
const taps: unknown[] = [];
158+
const device = createInteractionDevice(partialClipSnapshot, {
159+
tap: async (_context, point) => {
160+
taps.push(point);
161+
},
162+
});
163+
164+
await assert.rejects(
165+
() => device.interactions.press(selector('label=Cash'), { session: 'default' }),
166+
(error: unknown) => {
167+
const details = (error as { details?: Record<string, unknown> }).details;
168+
assert.equal(details?.reason, 'offscreen_selector');
169+
assert.equal(details?.scrollDirection, 'down');
170+
assert.match(String(details?.hint), /scroll down/i);
171+
// #1366 recovery must be bounded: a single large (fling) scroll overshoots,
172+
// so the hint steers to small steps / a bounded gesture pan.
173+
assert.match(String(details?.hint), /small steps/i);
174+
assert.match(String(details?.hint), /gesture pan/i);
124175
return true;
125176
},
126177
);

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,10 @@ import {
2424
resolveRefLabel,
2525
} from '../../../snapshot/snapshot-processing.ts';
2626
import {
27+
classifyOffscreenScrollDirection,
2728
isNodeVisibleOnScreen,
2829
resolveEffectiveViewportRect,
30+
type OffscreenScrollDirection,
2931
} from '../../../snapshot/mobile-snapshot-semantics.ts';
3032
import { containsPoint, resolveViewportRect } from '../../../utils/rect-visibility.ts';
3133
import { isSnapshotNodeInteractionBlocked } from '../../../snapshot/snapshot-occlusion.ts';
@@ -697,7 +699,14 @@ function assertVisibleSelectorTarget(
697699
throwIfOffscreenInteractionTarget(node, nodes, {
698700
message: `Selector ${selector} resolved to an off-screen element and is not safe to ${action}`,
699701
details: { reason: 'offscreen_selector', selector },
700-
hint: `The element is outside the visible viewport — likely inside a closed drawer, another tab, or scrolled content. Scroll toward it or open its container, take a fresh snapshot, then retry ${action}.`,
702+
// A selector re-resolves against a fresh snapshot on every attempt, so the
703+
// recovery is: move the named direction, then retry THIS selector — no
704+
// separate snapshot step, and no @ref (a scroll expires the ref frame,
705+
// #1366). Naming the direction stops the wrong-way / retry-the-same-ref loop;
706+
// bounded steps stop the overshoot loop — a single large scroll (fling
707+
// momentum on iOS) can sail past the target, so a short gesture pan lands it.
708+
hint: (direction) =>
709+
`${scrollRevealClause(direction)} in small steps, retrying ${action} with the same selector after each (it re-resolves against a fresh snapshot). A single large scroll can overshoot the target; a short bounded gesture pan lands it more reliably. If it is inside a closed drawer or another tab, open that container first.`,
701710
});
702711
}
703712

@@ -710,10 +719,23 @@ function assertVisibleRefTarget(
710719
throwIfOffscreenInteractionTarget(node, nodes, {
711720
message: `Ref ${refInput} is off-screen and not safe to ${action}`,
712721
details: { reason: 'offscreen_ref', ref: normalizeRef(refInput) },
713-
hint: `Use scroll with the direction from the off-screen summary, take a fresh snapshot, then retry ${action} with the new ref or a selector.`,
722+
// The scroll that reveals the target expires the ref frame (#1366, ADR
723+
// 0014), so retrying this @ref would be rejected next. Steer to a selector,
724+
// which re-resolves against a fresh snapshot and bypasses the ref-frame guard.
725+
hint: (direction) =>
726+
`${scrollRevealClause(direction)} in small steps (a single large scroll can overshoot; a short bounded gesture pan lands it more reliably), then retry ${action} with a selector (e.g. text=/id=) rather than this @ref — the scroll expires the ref frame, so re-run snapshot -i before reusing any @ref.`,
714727
});
715728
}
716729

730+
// Shared lead-in for both off-screen hints. Names the concrete `scroll <dir>`
731+
// when the geometry gives one, and falls back to the generic phrasing when the
732+
// target is off more than one edge in a way that has no single reveal. Callers
733+
// append the bounded-steps guidance: a single large scroll (fling momentum on
734+
// iOS) can sail past the target, so small bounded moves are what actually land.
735+
function scrollRevealClause(direction: OffscreenScrollDirection | null): string {
736+
return direction ? `Scroll ${direction} toward it` : 'Scroll toward it';
737+
}
738+
717739
/**
718740
* ADR 0011 native-ref preflight: `click @ref` / `fill @ref` fast paths
719741
* dispatch straight to `backend.tapTarget`/`fillTarget`, and a backend fast
@@ -766,14 +788,25 @@ export async function preflightNativeRefInteraction(
766788
function throwIfOffscreenInteractionTarget(
767789
node: SnapshotNode,
768790
nodes: SnapshotState['nodes'],
769-
failure: { message: string; details: Record<string, unknown>; hint: string },
791+
failure: {
792+
message: string;
793+
details: Record<string, unknown>;
794+
hint: (direction: OffscreenScrollDirection | null) => string;
795+
},
770796
): void {
771797
const viewport = node.rect ? resolveEffectiveViewportRect(node, nodes) : null;
772798
if (!node.rect || !viewport || isNodeVisibleOnScreen(node, nodes)) return;
799+
// The direction that scrolls this off-screen target into view. Named in the
800+
// hint (and surfaced as a machine-readable detail) so the recovery is a single
801+
// deterministic move instead of a guess (#1366). Derived from the same
802+
// boundary the rejection above used, so partial clips and off-screen
803+
// containers get a direction too, not just fully-scrolled-out items.
804+
const scrollDirection = classifyOffscreenScrollDirection(node, nodes);
773805
throw new AppError('COMMAND_FAILED', failure.message, {
774806
...failure.details,
775807
rect: node.rect,
776808
viewport,
777-
hint: failure.hint,
809+
...(scrollDirection ? { scrollDirection } : {}),
810+
hint: failure.hint(scrollDirection),
778811
});
779812
}

‎src/daemon/handlers/__tests__/interaction.test.ts‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2593,8 +2593,15 @@ test('press @ref fails fast when the target is off-screen', async () => {
25932593
if (response && !response.ok) {
25942594
expect(response.error.code).toBe('COMMAND_FAILED');
25952595
expect(response.error.message).toMatch(/off-screen/i);
2596-
expect(response.error.hint).toMatch(/scroll.*fresh snapshot/i);
2596+
// #1366: the hint names the concrete scroll direction, steers to a
2597+
// selector-based retry (a @ref would be rejected as expired after the scroll),
2598+
// and prescribes bounded movement (a large fling scroll overshoots).
2599+
expect(response.error.hint).toMatch(/scroll down/i);
2600+
expect(response.error.hint).toMatch(/selector/i);
2601+
expect(response.error.hint).toMatch(/small steps/i);
2602+
expect(response.error.hint).toMatch(/gesture pan/i);
25972603
expect(response.error.details?.reason).toBe('offscreen_ref');
2604+
expect(response.error.details?.scrollDirection).toBe('down');
25982605
}
25992606
});
26002607

@@ -2715,8 +2722,11 @@ test('fill @ref fails fast when the target is off-screen', async () => {
27152722
if (response && !response.ok) {
27162723
expect(response.error.code).toBe('COMMAND_FAILED');
27172724
expect(response.error.message).toMatch(/off-screen/i);
2718-
expect(response.error.hint).toMatch(/scroll.*fresh snapshot/i);
2725+
// #1366: direction-named, selector-first recovery hint (see press test above).
2726+
expect(response.error.hint).toMatch(/scroll down/i);
2727+
expect(response.error.hint).toMatch(/selector/i);
27192728
expect(response.error.details?.reason).toBe('offscreen_ref');
2729+
expect(response.error.details?.scrollDirection).toBe('down');
27202730
}
27212731
});
27222732

‎src/snapshot/mobile-snapshot-semantics.ts‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,82 @@ export function resolveEffectiveViewportRect(
150150
return resolveViewportRect(nodes, node.rect ?? { x: 0, y: 0, width: 0, height: 0 });
151151
}
152152

153+
/** The concrete `scroll <direction>` that brings an off-screen target into view. */
154+
export type OffscreenScrollDirection = 'up' | 'down' | 'left' | 'right';
155+
156+
/**
157+
* The direction to `scroll` so an off-screen interaction target comes into view.
158+
* Derived from the SAME two boundaries `isNodeVisibleOnScreen` rejects against,
159+
* so every rejected target yields a direction (#1366) — not just the subset a
160+
* rect-vs-one-viewport check catches:
161+
*
162+
* 1. The rect has no overlap with its effective (scroll-container) viewport —
163+
* an item scrolled out of an on-screen list. Direction from the container.
164+
* 2. The rect still overlaps its container, but the tap-point CENTER is pushed
165+
* outside the ROOT viewport — a child inside an off-screen drawer, or a row
166+
* straddling the viewport edge whose center is past it. The rect-vs-effective
167+
* -viewport form misses both; this reads the boundary that actually failed.
168+
*
169+
* Direction follows the reveal convention the CLI hints already use
170+
* (`cli-help.ts`): off-screen below -> scroll down, above -> up, horizontal
171+
* mirror; the axis with the largest center overshoot wins so a corner-off target
172+
* gets its dominant move. Returns null only when the node is on-screen.
173+
*/
174+
export function classifyOffscreenScrollDirection(
175+
node: Pick<SnapshotNode, 'rect' | 'index' | 'parentIndex' | 'type' | 'role' | 'subrole'>,
176+
nodes: SnapshotNode[],
177+
byIndex: Map<number, SnapshotNode> = buildSnapshotNodeMap(nodes),
178+
): OffscreenScrollDirection | null {
179+
if (!node.rect) {
180+
return null;
181+
}
182+
// Boundary 1: fully separated from the effective (scroll-container) viewport.
183+
const effectiveViewport = resolveEffectiveViewportRect(node, nodes, byIndex);
184+
if (effectiveViewport && !isRectVisibleInViewport(node.rect, effectiveViewport)) {
185+
const direction = directionOfCenterOutsideViewport(node.rect, effectiveViewport);
186+
if (direction) {
187+
return direction;
188+
}
189+
}
190+
// Boundary 2: tap-point center outside the root viewport (off-screen container
191+
// or an edge-straddling rect whose center is past the frame).
192+
const rootViewport = resolveViewportRect(nodes, node.rect);
193+
if (rootViewport && !isTapPointInsideViewport(node.rect, rootViewport)) {
194+
const direction = directionOfCenterOutsideViewport(node.rect, rootViewport);
195+
if (direction) {
196+
return direction;
197+
}
198+
}
199+
return null;
200+
}
201+
202+
/**
203+
* The dominant edge the rect's tap-point center sits beyond, or null when the
204+
* center is within the viewport on both axes. Uses the same unrounded center as
205+
* `isTapPointInsideViewport` so the direction agrees with the rejection at the
206+
* pixel boundary.
207+
*/
208+
function directionOfCenterOutsideViewport(
209+
rect: Rect,
210+
viewport: Rect,
211+
): OffscreenScrollDirection | null {
212+
const centerX = rect.x + rect.width / 2;
213+
const centerY = rect.y + rect.height / 2;
214+
const overshoots: Array<{ direction: OffscreenScrollDirection; amount: number }> = [
215+
{ direction: 'down', amount: centerY - (viewport.y + viewport.height) },
216+
{ direction: 'up', amount: viewport.y - centerY },
217+
{ direction: 'right', amount: centerX - (viewport.x + viewport.width) },
218+
{ direction: 'left', amount: viewport.x - centerX },
219+
];
220+
let best: { direction: OffscreenScrollDirection; amount: number } | null = null;
221+
for (const candidate of overshoots) {
222+
if (candidate.amount > 0 && (!best || candidate.amount > best.amount)) {
223+
best = candidate;
224+
}
225+
}
226+
return best?.direction ?? null;
227+
}
228+
153229
function deriveContainerHints(
154230
allNodes: SnapshotNode[],
155231
offscreenNodes: SnapshotNode[],

0 commit comments

Comments
 (0)