Skip to content

Commit 5a24030

Browse files
fix(ios): derive hittable from geometric actionability on the AX bridge
The snapshot capability table has declared `hittable = geometric-actionability` for every iOS backend (#1933), and the XCTest runner has always stamped it. When 0.21.0 made the host AX bridge the Simulator's snapshot source, the bridge reader mapped the traits word into `enabled` and `selected` but handed over no hit result and disclosed `hittability` as unavailable on every capture (#2638). A `hittable:` selector therefore matched nothing on the bridge that the runner would match. The reader now stamps `hittable` on its raw nodes from the node's own frame and the reported viewport, using a shared `isGeometricallyActionable` that is the TypeScript twin of the runner's Swift `SnapshotGeometry.isGeometricallyActionable`, `CGRect.contains` half-open edges included. It discloses `hittability` as unavailable only when no viewport is reported, keeping the residue owner the fact owner (#2199). Stamping the raw source rather than the fold makes `snapshot --raw` match too. This restores `hittable` parity between the two producers. It does not resolve the two native-stack Maestro flows that motivated the investigation: those are a separate behind-modal occlusion capture divergence, tracked apart. Co-Authored-By: Apex <[email protected]>
1 parent c32a09c commit 5a24030

11 files changed

Lines changed: 230 additions & 26 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,19 @@
22

33
## Unreleased
44

5+
- Fixed (ios): a local Simulator snapshot taken through the host AX bridge once again publishes the
6+
geometric `hittable` fact, so `is hittable` and a `hittable:` selector resolve the same controls on
7+
the bridge and the XCTest runner. The snapshot capability table has declared `hittable =
8+
geometric-actionability` for every backend (#1933), and the runner has always stamped it; when 0.21.0
9+
made the AX bridge the Simulator's snapshot source, the bridge reader mapped the traits word into
10+
`enabled` and `selected` but handed over no hit result and disclosed `hittability` as unavailable on
11+
every capture, so a `hittable:` selector matched nothing on the bridge that the runner would match.
12+
The reader now derives `hittable` on its raw nodes from the same rule the runner's Swift uses — the
13+
shared `isGeometricallyActionable`, `CGRect.contains` half-open edges and all — applied to the node's
14+
own frame and the reported viewport, and discloses `hittability` as unavailable only when no viewport
15+
is reported. Because it stamps the raw source rather than the fold, `snapshot --raw` matches too, and
16+
the residue owner stays the fact owner (#2199). Closes the hittable-absence half of the AX-bridge
17+
capture divergence (#2638).
518
- Fixed (ios): a local Simulator snapshot taken through the host AX bridge once again carries the
619
accessibility `selected` state, so `is selected`, a `selected=true` selector, and a Maestro
720
`selected:` qualifier match the active control. When 0.21.0 made the AX bridge the Simulator's

‎packages/capture-kit/src/ios-snapshot-acquisition.ts‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,12 @@ import type { RawSnapshotNode, Rect } from '@agent-device/kernel/snapshot';
1717
* this module derives.
1818
*
1919
* `apple-runner` and `simulator-ax-bridge` are absent by construction, not by omission: each
20-
* builds its own residue at the source (the bridge adapter emits `unavailable-fact: hittability`
21-
* on every capture; the runner presents and reports its own facts), so a row here would be a
22-
* declaration with no reader — which is how the table came to claim the bridge had hittability
23-
* evidence while every bridge-served `snapshot -i` printed the opposite (#2199). Keying the
24-
* record on {@link IosProviderAcquisitionProducer} makes that claim not compile.
20+
* builds its own residue at the source (the bridge reader derives geometric `hittable` onto its
21+
* nodes and emits `unavailable-fact: hittability` only when the viewport is missing; the runner
22+
* presents and reports its own facts), so a row here would be a declaration with no reader — which
23+
* is how the table once came to claim the bridge had hittability evidence while every bridge-served
24+
* `snapshot -i` printed the opposite (#2199). Keying the record on
25+
* {@link IosProviderAcquisitionProducer} makes that claim not compile.
2526
*/
2627
const IOS_PROVIDER_ACQUISITION_CAPABILITY_VALUES = {
2728
'appium-source': {

‎packages/capture-kit/src/ios-snapshot-engine/geometry-policy.ts‎

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { Rect, RawSnapshotNode } from '@agent-device/kernel/snapshot';
2-
import { containsPoint, isPositiveFiniteRect } from '@agent-device/kernel/rect';
2+
import { isPositiveFiniteRect } from '@agent-device/kernel/rect';
33
import { normalizeType } from '@agent-device/contracts/snapshot';
44
import type { IosSnapshotFoldPolicy } from './types.ts';
55

@@ -27,18 +27,6 @@ export type GeometryDecision = Readonly<{
2727
establishesScrollAnchor: boolean;
2828
}>;
2929

30-
export function isGeometricallyActionable(
31-
enabled: boolean,
32-
rect: Rect | undefined,
33-
viewport: Rect,
34-
): boolean {
35-
return Boolean(
36-
enabled &&
37-
isPositiveFiniteRect(rect) &&
38-
containsPoint(viewport, rect.x + rect.width / 2, rect.y + rect.height / 2),
39-
);
40-
}
41-
4230
export function rootTraversal(): TraversalState {
4331
return { projectedOut: false };
4432
}

‎packages/capture-kit/src/ios-snapshot-engine/geometry.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import type { Rect, RawSnapshotNode } from '@agent-device/kernel/snapshot';
2+
import { isGeometricallyActionable } from '@agent-device/kernel/rect';
23
import { validateIosSnapshotGraph } from './graph.ts';
34
import {
4-
isGeometricallyActionable,
55
rootTraversal,
66
traversalDecision,
77
type BranchState,

‎packages/contracts/src/ios-snapshot.ts‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,11 @@ export type IosSnapshotFact =
8484
* element tree — and deliberately not every {@link IosSnapshotProducer}. `apple-runner` and
8585
* `simulator-ax-bridge` build their own residue at the source, so a capability declared for
8686
* them here would be a claim nothing consults: exactly the shape that let the table say the
87-
* Simulator bridge had hittability evidence while the bridge adapter emitted
88-
* `unavailable-fact: hittability` on every capture (#2199). Narrowing the producer makes that
89-
* claim unrepresentable rather than merely wrong.
87+
* Simulator bridge had hittability evidence while the bridge adapter published none and emitted
88+
* `unavailable-fact: hittability` on every capture (#2199). The bridge now derives geometric
89+
* `hittable` at the source and drops that residue when its inputs are established, so a table row
90+
* for it would still have no reader. Narrowing the producer makes that claim unrepresentable
91+
* rather than merely wrong.
9092
*
9193
* Truncation is not a field here: it is the one fact the runner and the bridge also need
9294
* answered, so it has a single owner over all four producers instead

‎packages/kernel/src/rect.test.ts‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
import assert from 'node:assert/strict';
22
import { test } from 'vitest';
33
import type { Rect } from './snapshot.ts';
4-
import { containsPoint, isRectVisibleInViewport, pickLargestRect } from './rect.ts';
4+
import {
5+
containsPoint,
6+
isGeometricallyActionable,
7+
isRectVisibleInViewport,
8+
pickLargestRect,
9+
} from './rect.ts';
510

611
const VIEWPORT: Rect = { x: 0, y: 0, width: 300, height: 500 };
712

@@ -23,3 +28,55 @@ test('pickLargestRect selects by area and returns null for an empty list', () =>
2328
assert.deepEqual(pickLargestRect([{ x: 0, y: 0, width: 2, height: 2 }, VIEWPORT]), VIEWPORT);
2429
assert.equal(pickLargestRect([]), null);
2530
});
31+
32+
// These rows are the TypeScript twin of the runner's Swift `SnapshotGeometry.isGeometricallyActionable`
33+
// (asserted over randomized rects by the snapshot differential and over authored rects by
34+
// CoordinateSpaceTests.swift). They pin it so the host AX bridge cannot drift from the XCTest runner.
35+
test('isGeometricallyActionable matches the Swift runner predicate', () => {
36+
assert.equal(
37+
isGeometricallyActionable(true, { x: 10, y: 10, width: 40, height: 40 }, VIEWPORT),
38+
true,
39+
);
40+
// Enabled but centered off-screen, or zero/negative/absent, is never actionable.
41+
assert.equal(
42+
isGeometricallyActionable(true, { x: 320, y: 10, width: 40, height: 40 }, VIEWPORT),
43+
false,
44+
);
45+
assert.equal(
46+
isGeometricallyActionable(true, { x: 10, y: 600, width: 40, height: 40 }, VIEWPORT),
47+
false,
48+
);
49+
assert.equal(
50+
isGeometricallyActionable(true, { x: 10, y: 10, width: 0, height: 40 }, VIEWPORT),
51+
false,
52+
);
53+
assert.equal(isGeometricallyActionable(true, undefined, VIEWPORT), false);
54+
// A disabled node with a centred frame is not actionable.
55+
assert.equal(
56+
isGeometricallyActionable(false, { x: 10, y: 10, width: 40, height: 40 }, VIEWPORT),
57+
false,
58+
);
59+
// CGRect.contains is half-open: the top/left edge counts, the right/bottom edge does not — the
60+
// divergence the shared `containsPoint` (inclusive) would otherwise hide from a `hittable:` selector.
61+
const topCentered: Rect = { x: 0, y: 0, width: 2, height: 2 };
62+
assert.equal(
63+
isGeometricallyActionable(true, topCentered, VIEWPORT),
64+
true,
65+
'center at the origin counts',
66+
);
67+
assert.equal(
68+
containsPoint(VIEWPORT, 300, 250),
69+
true,
70+
'containsPoint stays inclusive on the right edge',
71+
);
72+
assert.equal(
73+
isGeometricallyActionable(true, { x: 299, y: 249, width: 2, height: 2 }, VIEWPORT),
74+
false,
75+
'a center on the right edge is not hittable',
76+
);
77+
assert.equal(
78+
isGeometricallyActionable(true, { x: 149, y: 499, width: 2, height: 2 }, VIEWPORT),
79+
false,
80+
'a center on the bottom edge is not hittable',
81+
);
82+
});

‎packages/kernel/src/rect.ts‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,32 @@ export function containsPoint(rect: Rect, x: number, y: number): boolean {
2727
return x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height;
2828
}
2929

30+
/**
31+
* The shared `hittable` predicate every iOS snapshot producer publishes (#1933): an enabled node
32+
* with a positive frame whose center falls inside the viewport. It is the TypeScript twin of the
33+
* runner's Swift `SnapshotGeometry.isGeometricallyActionable`, including `CGRect.contains`'s
34+
* half-open right/bottom edges — a center landing exactly on the viewport's right or bottom edge is
35+
* not hittable on either producer. The host AX bridge derives the source bit from the node's own
36+
* frame and the fold intersects it with the clipped frame, so a `hittable:` selector cannot tell the
37+
* two producers apart. Kept here so both packages read one definition rather than each re-encoding
38+
* the rule.
39+
*/
40+
export function isGeometricallyActionable(
41+
enabled: boolean,
42+
rect: Rect | undefined,
43+
viewport: Rect,
44+
): boolean {
45+
if (!enabled || !isPositiveFiniteRect(rect)) return false;
46+
const centerX = rect.x + rect.width / 2;
47+
const centerY = rect.y + rect.height / 2;
48+
return (
49+
centerX >= viewport.x &&
50+
centerX < viewport.x + viewport.width &&
51+
centerY >= viewport.y &&
52+
centerY < viewport.y + viewport.height
53+
);
54+
}
55+
3056
export function pickLargestRect(rects: readonly Rect[]): Rect | null {
3157
let best: Rect | null = null;
3258
let bestArea = -1;

‎packages/platform-apple/src/snapshot-source/adapter.test.ts‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,27 @@ test('the Simulator AX source returns raw acquisition facts and discloses unsupp
6666
targetId: 'target-1',
6767
generation: 'generation-1',
6868
});
69+
// With the viewport reported the reader owns `hittable` and stamps it on every node, so hittability
70+
// is not disclosed as unavailable; only the interactive-query facet it cannot honour is.
6971
assert.deepEqual(result.acquisition.residue, [
72+
{ kind: 'unavailable-fact', fact: 'interactive-query' },
73+
]);
74+
assert.equal(result.acquisition.nodes[0]?.hittable, false, 'the stamped root is not hittable');
75+
76+
// The complement (#2199: the residue owner is the fact owner): strip the viewport and the reader
77+
// publishes no `hittable` claim and discloses hittability as unavailable again.
78+
fixture.omitViewport = true;
79+
const viewportless = await source.acquire({
80+
target: { ...sourceTarget, targetId: 'target-1' },
81+
hint,
82+
});
83+
fixture.omitViewport = false;
84+
assert.equal(viewportless.stage, 'acquired');
85+
assert.equal(viewportless.acquisition.nodes[0]?.hittable, undefined);
86+
assert.deepEqual(viewportless.acquisition.residue, [
7087
{ kind: 'unavailable-fact', fact: 'hittability' },
7188
{ kind: 'unavailable-fact', fact: 'interactive-query' },
89+
{ kind: 'missing-viewport', reason: 'not-provided' },
7290
]);
7391

7492
const regularDepthOne = await source.acquire({
@@ -443,6 +461,8 @@ type AdapterFixture = {
443461
remoteContent: boolean;
444462
/** Whether the fake guest's tree holds a window reporting the app box quarter-turned (#2612). */
445463
turnedWindow: boolean;
464+
/** Whether the fake guest's root omits its frame, leaving the capture without a viewport. */
465+
omitViewport: boolean;
446466
omitRecovery: boolean;
447467
diagnostics: Record<string, unknown>[];
448468
};
@@ -469,6 +489,7 @@ function createAdapterHost(buildDelayMs = 0): AdapterFixture {
469489
malformedTree: false,
470490
remoteContent: false,
471491
turnedWindow: false,
492+
omitViewport: false,
472493
omitRecovery: false,
473494
diagnostics: [],
474495
};
@@ -542,6 +563,11 @@ class AdapterSocket extends EventEmitter implements SnapshotSourceSocket {
542563
this.fixture = fixture;
543564
}
544565

566+
/** The guest root's frame, or undefined when the fixture reports a viewportless tree. */
567+
private rootFrame(): Record<string, number> | undefined {
568+
return this.fixture.omitViewport ? undefined : { X: 0, Y: 0, Width: 390, Height: 844 };
569+
}
570+
545571
write(frame: Buffer): boolean {
546572
const bodyLength = frame.readUInt32BE(0);
547573
const request = JSON.parse(frame.subarray(4, bodyLength + 4).toString('utf8')) as {
@@ -610,7 +636,7 @@ class AdapterSocket extends EventEmitter implements SnapshotSourceSocket {
610636
? null
611637
: {
612638
XC_kAXXCAttributeElementType: 'Application',
613-
XC_kAXXCAttributeFrame: { X: 0, Y: 0, Width: 390, Height: 844 },
639+
XC_kAXXCAttributeFrame: this.rootFrame(),
614640
XC_kAXXCAttributeChildren:
615641
request.maxDepth === 1
616642
? [{ XC_kAXXCAttributeElementType: 'Button', XC_kAXXCAttributeChildren: [] }]

‎packages/platform-apple/src/snapshot-source/adapter.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,8 +261,16 @@ function createAcquisitionResidue(
261261
truncated: boolean,
262262
viewport: IosViewportEvidence,
263263
) {
264+
// With a reported viewport the bridge reader has already stamped geometric `hittable` onto every
265+
// node (it owns that fact and the residue), so it is not declared unavailable. Only when the
266+
// viewport is missing does the reader withhold `hittable` and disclose it as an unavailable fact —
267+
// the same "the residue owner is the fact owner" rule that keeps geometry from minting the claim.
268+
const hittabilityResidue =
269+
viewport.kind === 'reported'
270+
? []
271+
: ([{ kind: 'unavailable-fact', fact: 'hittability' }] as const);
264272
return Object.freeze([
265-
{ kind: 'unavailable-fact', fact: 'hittability' } as const,
273+
...hittabilityResidue,
266274
...(hint.interactiveOnly
267275
? ([{ kind: 'unavailable-fact', fact: 'interactive-query' }] as const)
268276
: []),

‎packages/platform-apple/src/snapshot-source/tree.test.ts‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ test('the bridge tree becomes one depth-first raw snapshot with viewport evidenc
5252
role: 'Application',
5353
rect: { x: 0, y: 0, width: 390, height: 844 },
5454
depth: 0,
55+
hittable: false,
5556
},
5657
{
5758
index: 1,
@@ -61,6 +62,7 @@ test('the bridge tree becomes one depth-first raw snapshot with viewport evidenc
6162
subrole: 'UIWindow',
6263
rect: { x: 0, y: 0, width: 390, height: 844 },
6364
depth: 1,
65+
hittable: true,
6466
},
6567
{
6668
index: 2,
@@ -69,6 +71,7 @@ test('the bridge tree becomes one depth-first raw snapshot with viewport evidenc
6971
label: 'Continue',
7072
rect: { x: 20, y: 700, width: 120, height: 48 },
7173
depth: 2,
74+
hittable: true,
7275
},
7376
]);
7477
assert.deepEqual(result.viewport, {
@@ -79,6 +82,59 @@ test('the bridge tree becomes one depth-first raw snapshot with viewport evidenc
7982
assert.equal(result.opaqueRemoteElements, 0);
8083
});
8184

85+
test('the bridge reader stamps geometric hittable onto every raw node, matching the runner', () => {
86+
const traitsWord = (bits: bigint) => bits.toString();
87+
const button = (text: string, rect: Record<string, number>, traitBits = 1n) => ({
88+
[automationType]: 9,
89+
[label]: text,
90+
[frame]: rect,
91+
[traits]: traitsWord(traitBits),
92+
[children]: [],
93+
});
94+
const decode = (kids: unknown[]) =>
95+
decodeSnapshotBridgeTree(
96+
{
97+
[application]: 'Application',
98+
[frame]: { X: 0, Y: 0, Width: 390, Height: 844 },
99+
[children]: kids,
100+
},
101+
{ truncated: false },
102+
limits,
103+
).nodes;
104+
const hittableOf = (kids: unknown[]) => decode(kids).map((node) => node.hittable);
105+
const notEnabled = 1n << 8n;
106+
107+
// A root has no parent to hit through, so it is published not-hittable exactly as the runner does.
108+
assert.deepEqual(hittableOf([]), [false], 'root is not hittable');
109+
// On-screen enabled is hittable; the same frame disabled is not; a frame centred below the
110+
// fold is not, even while enabled. These are the Swift `normalized()` cases pinned in
111+
// CoordinateSpaceTests.swift, replayed against the bridge reader.
112+
assert.deepEqual(
113+
hittableOf([
114+
button('Continue', { X: 20, Y: 700, Width: 120, Height: 48 }),
115+
button('Place order', { X: 20, Y: 600, Width: 120, Height: 48 }, 1n | notEnabled),
116+
button('Offscreen', { X: 20, Y: 2000, Width: 120, Height: 48 }),
117+
]),
118+
[false, true, false, false],
119+
);
120+
121+
// Without a reported viewport the reader has no rule input, so it publishes no hittable claim at
122+
// all rather than guessing — the fold keeps hittability withheld until the runner serves the frame.
123+
const noViewport = decodeSnapshotBridgeTree(
124+
{
125+
[application]: 'Application',
126+
[children]: [button('Continue', { X: 20, Y: 700, Width: 120, Height: 48 })],
127+
},
128+
{ truncated: false },
129+
limits,
130+
).nodes;
131+
assert.deepEqual(
132+
noViewport.map((node) => node.hittable),
133+
[undefined, undefined],
134+
'a missing viewport publishes no hittable claim',
135+
);
136+
});
137+
82138
test('the bridge tree counts web-hosted remote leaves that reach the viewport', () => {
83139
const viewport = { X: 0, Y: 0, Width: 390, Height: 844 };
84140
const remoteLeaf = (rect?: Record<string, number>) => ({

0 commit comments

Comments
 (0)