Skip to content

Commit 47b1cae

Browse files
authored
fix(ios): refuse Simulator bridge trees that end at a web view's remote content (#2484) (#2486)
* 0.21.1 * fix(ios): refuse Simulator bridge trees that end at a web view's remote content (#2484) Since 0.21.0 the host AX bridge is the snapshot source for local iOS Simulators. It reads one process, and a WebKit page lives in another: Safari and WKWebView screens were published as chrome plus empty webview nodes, with no ref reaching the page. The decoder now counts AXRemoteElement leaves that sit under a WebView ancestor and reach the viewport, and the source refuses such a tree as remote-content-boundary. The existing route fallback serves XCTest, which resolves remote elements, for the rest of the app generation and discloses the switch in the snapshot warning. Frameless leaves are refused; zero-area and off-screen ones are published. Adds a fixture-backed smoke scenario that drives the WebView lab through the default route, amends ADR 0004 and the bridge README, and shares the e2e snapshotNodes helper.
1 parent df0a0f7 commit 47b1cae

16 files changed

Lines changed: 345 additions & 30 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22

33
## Unreleased
44

5+
- Fixed: iOS Simulator snapshots of Safari and of apps with a `WKWebView` stopped showing the
6+
page in 0.21.0 — chrome plus empty `[webview]` nodes, no links, text, or form fields, so no ref
7+
could reach the page (#2484). The host AX bridge that 0.21.0 made the Simulator's snapshot source
8+
reads one process, and WebKit content lives in another; the bridge delivered the boundary as
9+
an `AXRemoteElement` leaf and the tree was published as if that were the screen. The source now
10+
refuses a tree whose on-screen web view ends at that leaf (`remote-content-boundary`) and the
11+
route serves XCTest, which resolves remote elements, for the rest of that app generation — the
12+
same path 0.20.x used. The snapshot discloses the switch through its warning, and a relaunch
13+
re-enables the bridge.
514
- Fixed: JUnit reports remain readable when replay results contain characters forbidden by XML 1.0,
615
replacing them with U+FFFD while preserving legal Unicode and whitespace. Original suite values
716
remain available in JSON and other reporters.

‎apple/snapshot-bridge/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,16 @@ resolution. Secondary owners such as the return-to-app status-bar control do
4848
not replace the native primary owner. The route's generation circuit remains
4949
disabled after fallback until that app relaunches.
5050

51+
## Remote content
52+
53+
The reader snapshots one process. A WebKit page — Safari's or a `WKWebView`'s —
54+
lives in a WebContent process and appears in that tree as an `AXRemoteElement`
55+
leaf under the web view. The guest returns the leaf as delivered; the host
56+
refuses a tree in which such a leaf sits under a web view and reaches the
57+
viewport, or reports no frame (`remote-content-boundary`), and routes that app
58+
generation to XCTest, which resolves remote elements (#2484). A zero-area or
59+
off-screen leaf is published: it hosts nothing the capture can miss.
60+
5161
## Bounded depth recovery
5262

5363
A healthy capture uses one native request. If native acquisition rejects it,

‎docs/adr/0004-ios-snapshot-backend-strategy.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,22 @@ fallback when bridge acquisition or presentation fails. Disable the bridge for t
4242
after fallback; a new app generation re-enables it. Physical devices, providers, custom-action
4343
captures, and interactions remain on their existing owners.
4444

45+
A WebKit page — Safari's, or a `WKWebView`'s — lives in a WebContent process and reaches UIKit's
46+
tree as an `AXRemoteElement` under the web view, with its children in that other process. The
47+
bridge reads one process, so it delivers the element as a leaf. The source refuses a tree in which
48+
such a leaf sits under a `WebView`-typed ancestor and reaches the viewport (`remote-content-boundary`)
49+
instead of publishing a screen without its page: refs issued from it would target the host views
50+
around the page rather than the page. A leaf whose frame is zero-area or off screen hosts nothing
51+
the capture can miss and is published; one that reports no frame is refused, because nothing proves
52+
it empty. Remote elements outside a web view are not classified — no capture has shown one — and a
53+
web view truncated away by the node or depth cap stays disclosed as truncation. XCTest resolves
54+
remote elements, so the fallback serves the page (#2484).
55+
56+
The refusal opens the generation circuit like any other bridge failure, so a hybrid app that showed
57+
one web screen takes XCTest for its remaining native screens until it relaunches — the 0.20.x path
58+
for every screen. Re-asking the bridge per capture would instead charge a refused bridge round trip
59+
to every `wait` poll on the web screen; the circuit keeps that cost to one capture per generation.
60+
4561
Keep the two public snapshot strategies explicit:
4662

4763
- **Regular visible strategy**: use recursive XCTest snapshots, emit the effective user-visible

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "agent-device",
3-
"version": "0.21.0",
3+
"version": "0.21.1",
44
"description": "Mobile app automation and verification for AI coding agents. CLI, MCP server, and typed Node.js API for iOS, Android, HarmonyOS, TV, web, macOS, and Linux.",
55
"mcpName": "io.github.callstack/agent-device",
66
"license": "MIT",

‎packages/platform-apple/src/snapshot-source/adapter.test.ts‎

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,56 @@ test('the Simulator AX source returns raw acquisition facts and discloses unsupp
9898
}
9999
});
100100

101+
test('the Simulator AX source refuses a tree that ends at content another process owns', async () => {
102+
const root = await mkdtemp(path.join(os.tmpdir(), 'agent-device-snapshot-adapter-remote-'));
103+
const sourceRoot = path.join(root, 'source');
104+
await (await import('@agent-device/host-kit/host-file')).ensureHostDirectory(sourceRoot);
105+
for (const name of [
106+
'SnapshotBridge.m',
107+
'SnapshotBridgeRuntime.m',
108+
'SnapshotBridgeRuntime.h',
109+
'SnapshotBridgeCapture.h',
110+
'SnapshotBridgeCapture.m',
111+
]) {
112+
await writeFile(path.join(sourceRoot, name), 'native source');
113+
}
114+
const fixture = createAdapterHost();
115+
fixture.remoteContent = true;
116+
const source = createSimulatorSnapshotSource({
117+
host: fixture.host,
118+
sourceRoot,
119+
cacheRoot: path.join(root, 'cache'),
120+
});
121+
const target = { ...targetForTest(), generation: 'generation-1', targetId: 'target-1' };
122+
123+
try {
124+
for (const request of [
125+
createIosSnapshotRequest(),
126+
createIosSnapshotRequest({ acquisitionIntent: 'surface-observation' }),
127+
]) {
128+
const outcome = await source.acquire({ target, hint: deriveIosCaptureHint(request) });
129+
assert.equal(outcome.stage, 'failed');
130+
if (outcome.stage === 'failed') {
131+
assert.equal(outcome.failure.kind, 'unsupported');
132+
assert.equal(outcome.failure.code, 'remote-content-boundary');
133+
assert.equal(outcome.failure.details?.remoteElements, 1);
134+
}
135+
}
136+
// A refused tree teaches no depth hint: nothing about it validated.
137+
assert.deepEqual(fixture.diagnostics, []);
138+
139+
fixture.remoteContent = false;
140+
const recovered = await source.acquire({
141+
target,
142+
hint: deriveIosCaptureHint(createIosSnapshotRequest()),
143+
});
144+
assert.equal(recovered.stage, 'acquired');
145+
} finally {
146+
await source.close();
147+
await rm(root, { recursive: true, force: true });
148+
}
149+
});
150+
101151
test('preparation consumes the same acquisition deadline as bridge I/O', async () => {
102152
const root = await mkdtemp(path.join(os.tmpdir(), 'agent-device-snapshot-adapter-deadline-'));
103153
const sourceRoot = path.join(root, 'source');
@@ -373,6 +423,8 @@ type AdapterFixture = {
373423
truncated: boolean;
374424
/** Whether the fake guest answers with sound counters but an unusable tree. */
375425
malformedTree: boolean;
426+
/** Whether the fake guest's tree ends at another process's content (a remote element leaf). */
427+
remoteContent: boolean;
376428
omitRecovery: boolean;
377429
diagnostics: Record<string, unknown>[];
378430
};
@@ -397,6 +449,7 @@ function createAdapterHost(buildDelayMs = 0): AdapterFixture {
397449
rejectLevelsAbove: undefined,
398450
truncated: false,
399451
malformedTree: false,
452+
remoteContent: false,
400453
omitRecovery: false,
401454
diagnostics: [],
402455
};
@@ -542,7 +595,22 @@ class AdapterSocket extends EventEmitter implements SnapshotSourceSocket {
542595
XC_kAXXCAttributeChildren:
543596
request.maxDepth === 1
544597
? [{ XC_kAXXCAttributeElementType: 'Button', XC_kAXXCAttributeChildren: [] }]
545-
: [],
598+
: this.fixture.remoteContent
599+
? [
600+
{
601+
XC_kAXXCAttributeElementType: 'WebView',
602+
XC_kAXXCAttributeFrame: { X: 0, Y: 0, Width: 390, Height: 844 },
603+
XC_kAXXCAttributeChildren: [
604+
{
605+
XC_kAXXCAttributeElementType: 'AXRemoteElement',
606+
XC_kAXXCAttributeElementBaseType: 'NSObject',
607+
XC_kAXXCAttributeFrame: { X: 0, Y: 0, Width: 390, Height: 844 },
608+
XC_kAXXCAttributeChildren: [],
609+
},
610+
],
611+
},
612+
]
613+
: [],
546614
},
547615
},
548616
{

‎packages/platform-apple/src/snapshot-source/adapter.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,15 @@ function createAcquisition(
231231
if (typeof generation !== 'string' || !generation) {
232232
throw snapshotSourceError('malformed-tree', 'generation-invalid');
233233
}
234+
// A tree that ends at a web view's out-of-process page would present the screen without the
235+
// page, and refs issued from it would target the host views around it rather than the page.
236+
// The source refuses it as a screen it cannot describe, like a missing automation mode, so the
237+
// route serves the XCTest runner, which resolves remote elements (#2484).
238+
if (decoded.opaqueRemoteElements > 0) {
239+
throw snapshotSourceError('unsupported', 'remote-content-boundary', {
240+
remoteElements: decoded.opaqueRemoteElements,
241+
});
242+
}
234243
const nodes = Object.freeze(
235244
decoded.nodes.map((node) => Object.freeze({ ...node, pid: target.pid })),
236245
);

‎packages/platform-apple/src/snapshot-source/tree.test.ts‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,74 @@ test('the bridge tree becomes one depth-first raw snapshot with viewport evidenc
7474
rect: { x: 0, y: 0, width: 390, height: 844 },
7575
});
7676
assert.equal(result.maxTraversalDepth, 2);
77+
assert.equal(result.opaqueRemoteElements, 0);
78+
});
79+
80+
test('the bridge tree counts web-hosted remote leaves that reach the viewport', () => {
81+
const viewport = { X: 0, Y: 0, Width: 390, Height: 844 };
82+
const remoteLeaf = (rect?: Record<string, number>) => ({
83+
[application]: 'AXRemoteElement',
84+
[baseType]: 'NSObject',
85+
...(rect ? { [frame]: rect } : {}),
86+
[children]: [],
87+
});
88+
const decode = (host: Record<string, unknown>, root: Record<string, unknown> = {}) =>
89+
decodeSnapshotBridgeTree(
90+
{ [application]: 'Application', [frame]: viewport, [children]: [host], ...root },
91+
{ truncated: false },
92+
limits,
93+
);
94+
const webView = (content: Record<string, unknown>) => ({
95+
[automationType]: 58,
96+
[frame]: viewport,
97+
[children]: [
98+
{
99+
[automationType]: 58,
100+
[baseType]: 'WKContentView',
101+
[frame]: viewport,
102+
[children]: [content],
103+
},
104+
],
105+
});
106+
107+
const opaque = decode(webView(remoteLeaf(viewport)));
108+
assert.equal(opaque.nodes[2]?.type, 'WebView');
109+
assert.equal(opaque.nodes[3]?.role, 'AXRemoteElement');
110+
assert.equal(opaque.opaqueRemoteElements, 1);
111+
112+
assert.equal(decode(webView(remoteLeaf())).opaqueRemoteElements, 1, 'frameless leaf refuses');
113+
assert.equal(
114+
decode(webView(remoteLeaf({ X: 0, Y: 0, Width: 0, Height: 0 }))).opaqueRemoteElements,
115+
0,
116+
'zero-area leaf hosts nothing',
117+
);
118+
assert.equal(
119+
decode(webView(remoteLeaf({ X: 0, Y: 2000, Width: 390, Height: 600 }))).opaqueRemoteElements,
120+
0,
121+
'off-screen leaf is not on this screen',
122+
);
123+
assert.equal(
124+
decode(webView(remoteLeaf({ X: 0, Y: 2000, Width: 390, Height: 600 })), { [frame]: undefined })
125+
.opaqueRemoteElements,
126+
1,
127+
'without a viewport a positive-area leaf refuses',
128+
);
129+
assert.equal(
130+
decode({ [automationType]: 0, [frame]: viewport, [children]: [remoteLeaf(viewport)] })
131+
.opaqueRemoteElements,
132+
0,
133+
'a remote leaf outside a web view is not classified',
134+
);
135+
assert.equal(
136+
decode(
137+
webView({
138+
...remoteLeaf(viewport),
139+
[children]: [{ [automationType]: 42, [label]: 'More information', [children]: [] }],
140+
}),
141+
).opaqueRemoteElements,
142+
0,
143+
'a crossed boundary is not opaque',
144+
);
77145
});
78146

79147
test('the bridge tree rejects unknown fields, invalid frames, and bounded overflows', () => {

‎packages/platform-apple/src/snapshot-source/tree.ts‎

Lines changed: 43 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { isPositiveFiniteRect } from '@agent-device/kernel/rect';
1+
import { isPositiveFiniteRect, isRectVisibleInViewport } from '@agent-device/kernel/rect';
22
import type { RawSnapshotNode, Rect } from '@agent-device/kernel/snapshot';
33
import type { IosViewportEvidence } from '@agent-device/contracts/ios-snapshot';
44
import { snapshotSourceError } from './errors.ts';
@@ -110,6 +110,20 @@ const CLASS_PROMOTED_TYPES: Readonly<Record<string, string>> = {
110110

111111
const NODE_KEYS = new Set<string>(Object.values(ATTRIBUTE));
112112

113+
/**
114+
* A WebKit page — Safari's, or a `WKWebView`'s — lives in a WebContent process and reaches UIKit's
115+
* tree as an `AXRemoteElement` under the web view, with its children in that other process. The
116+
* guest reader snapshots one process, so it delivers that element as a leaf (#2484). Such a leaf
117+
* is opaque when it sits under a `WebView`-typed ancestor and its frame reaches the viewport: the
118+
* page is on screen and the tree does not describe it. A leaf whose frame is zero-area or off
119+
* screen hosts nothing the capture can miss; one that reports no frame at all is refused, because
120+
* nothing proves it is empty. Remote elements outside a web view are not classified here — no
121+
* capture has shown one — and content truncated away above the web view stays disclosed as
122+
* truncation, not as a boundary.
123+
*/
124+
const REMOTE_ELEMENT_CLASS = 'AXRemoteElement';
125+
const WEB_VIEW_TYPE = 'WebView';
126+
113127
export function decodeSnapshotBridgeTree(
114128
tree: unknown,
115129
envelope: Readonly<{ truncated: unknown }>,
@@ -120,9 +134,10 @@ export function decodeSnapshotBridgeTree(
120134
throw snapshotSourceError('malformed-tree', 'guest-tree-root-invalid');
121135
}
122136
const nodes: RawSnapshotNode[] = [];
137+
const webHostedRemoteLeaves: (Rect | undefined)[] = [];
123138
let maxTraversalDepth = 0;
124139
for (const root of roots) {
125-
visitNode(root, undefined, 0);
140+
visitNode(root, undefined, 0, false);
126141
}
127142
if (nodes.length > limits.maxNodes) {
128143
throw snapshotSourceError('malformed-tree', 'node-limit-exceeded', {
@@ -139,18 +154,22 @@ export function decodeSnapshotBridgeTree(
139154
if (typeof envelope.truncated !== 'boolean') {
140155
throw snapshotSourceError('malformed-tree', 'truncated-invalid');
141156
}
157+
const viewport = viewportFromRoot(
158+
nodes.find((node) => node.type === 'Application' || node.type === 'Window'),
159+
);
142160
return {
143161
nodes,
144162
maxTraversalDepth,
145-
viewport: viewportFromRoot(
146-
nodes.find((node) => node.type === 'Application' || node.type === 'Window'),
147-
),
163+
viewport,
164+
opaqueRemoteElements: webHostedRemoteLeaves.filter((rect) => isOpaqueRemoteLeaf(rect, viewport))
165+
.length,
148166
};
149167

150168
function visitNode(
151169
value: Record<string, unknown>,
152170
parentIndex: number | undefined,
153171
depth: number,
172+
underWebView: boolean,
154173
): void {
155174
if (nodes.length >= limits.maxNodes) {
156175
throw snapshotSourceError('malformed-tree', 'node-limit-exceeded', {
@@ -170,9 +189,13 @@ export function decodeSnapshotBridgeTree(
170189
const node = nodeFacts(value, index, parentIndex, depth);
171190
nodes.push(node);
172191
maxTraversalDepth = Math.max(maxTraversalDepth, depth);
192+
if (isWebHostedRemoteLeaf(node, children.length, underWebView)) {
193+
webHostedRemoteLeaves.push(node.rect);
194+
}
195+
const hostsWeb = underWebView || node.type === WEB_VIEW_TYPE;
173196
for (const child of children) {
174197
if (!isRecord(child)) throw snapshotSourceError('malformed-tree', 'child-invalid');
175-
visitNode(child, index, depth + 1);
198+
visitNode(child, index, depth + 1, hostsWeb);
176199
}
177200
}
178201
}
@@ -224,6 +247,20 @@ function elementTypeName(
224247
return ELEMENT_TYPE_NAMES[automationType] ?? 'Other';
225248
}
226249

250+
function isWebHostedRemoteLeaf(
251+
node: RawSnapshotNode,
252+
childCount: number,
253+
underWebView: boolean,
254+
): boolean {
255+
return underWebView && node.role === REMOTE_ELEMENT_CLASS && childCount === 0;
256+
}
257+
258+
function isOpaqueRemoteLeaf(rect: Rect | undefined, viewport: IosViewportEvidence): boolean {
259+
if (rect === undefined) return true;
260+
if (!isPositiveFiniteRect(rect)) return false;
261+
return viewport.kind !== 'reported' || isRectVisibleInViewport(rect, viewport.rect);
262+
}
263+
227264
function frameFromGuest(value: unknown): Rect | undefined {
228265
if (value === undefined) return undefined;
229266
if (!isRecord(value)) throw snapshotSourceError('malformed-tree', 'frame-invalid');

‎packages/platform-apple/src/snapshot-source/types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,4 +131,9 @@ export type SnapshotSourceDecodedTree = Readonly<{
131131
nodes: readonly RawSnapshotNode[];
132132
viewport: IosViewportEvidence;
133133
maxTraversalDepth: number;
134+
/**
135+
* `AXRemoteElement` leaves under a web view whose frame reaches the viewport, or that report no
136+
* frame: pages the reader could not cross into (#2484).
137+
*/
138+
opaqueRemoteElements: number;
134139
}>;

‎server.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,12 @@
77
"url": "https://github.com/callstack/agent-device",
88
"source": "github"
99
},
10-
"version": "0.21.0",
10+
"version": "0.21.1",
1111
"packages": [
1212
{
1313
"registryType": "npm",
1414
"identifier": "agent-device",
15-
"version": "0.21.0",
15+
"version": "0.21.1",
1616
"transport": {
1717
"type": "stdio"
1818
},

0 commit comments

Comments
 (0)