Skip to content

Commit 066b376

Browse files
KrzysztofMochKrzysztof Moch
andauthored
fix(ios): read simulator snapshots past 64 accessibility levels (#4089)
* fix(ios): read simulator snapshots past 64 accessibility levels The simulator AX bridge cut every regular capture at 65 levels (maxTraversalDepth 64), so a control nested deeper was missing from the snapshot and press by label or ref failed, while a coordinate press still worked. The bridge already reads past the AX server's per-request limit through continuations; only the daemon's default bound stopped it. Raise the default to the bridge's own bound (128). Each native request still reads at most 65 levels, so trees up to 65 levels issue exactly the same requests, rejections, and depth hints as before. Double the continuation budget to 64 requests, since each root-to-leaf path needs one continuation per accepted-depth window and the bound doubled. * fix(ios): keep the full requested depth in the recovery diagnostic Depth hints now cap their comparison at the first request themselves, so the adapter passes the requested levels unchanged and ios_snapshot_source_recovery reports 129 for a 128-level capture. Correct the per-request limit notes and the changeset's budget claim. --------- Co-authored-by: Krzysztof Moch <[email protected]>
1 parent 0022611 commit 066b376

8 files changed

Lines changed: 119 additions & 31 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'agent-device': patch
3+
---
4+
5+
Fix iOS simulator snapshots dropping controls nested more than 64 accessibility levels deep, so `press` by label or ref could not find them. Regular snapshots now read up to 128 levels: each native request still reads at most 65 levels, and continuations deliver the rest. The continuation budget grows from 32 to 64 requests to cover the extra levels, so fewer wide deep screens exhaust it and fall back to the slower runner capture; a capture that still exhausts it fails closed and falls back as before.

‎apple/snapshot-bridge/README.md‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -78,16 +78,20 @@ zero-area or off-screen leaf is published: it hosts nothing the capture can miss
7878
A healthy capture uses one native request. If native acquisition rejects it,
7979
`SnapshotBridgeCapture.m` retries supported native failure codes at lower depths
8080
and fetches withheld children from their accessibility elements. The completed
81-
tree keeps the original depth and node limits; partial trees disclose truncation.
82-
The traversal depth counts edges below the root; native requests count the root
83-
as one level. Each acquisition allows two lower-depth retries, and recovery
84-
allows at most 32 native requests within the existing capture deadline,
85-
checks foreground ownership on every request, and returns a failure when it
86-
cannot complete a continuation. Budget exhaustion and malformed continuations
87-
use non-launch failure codes, so the route falls back without launch re-polling.
88-
At each native fragment boundary, an absent or invalid child count means unknown
89-
completeness and fails closed. Natural leaves above that boundary need no
90-
continuation evidence. Unchanged native dictionaries and child arrays are reused.
81+
tree keeps the original depth and node limits; partial trees disclose
82+
truncation. The traversal depth counts edges below the root; native requests
83+
count the root as one level. One native request reads at most 65 levels; deeper
84+
levels, up to the 128-level bound, arrive through continuations. On deep trees
85+
the AX server rejects a request that would return more than about 62 levels, so
86+
there the first request is rejected once and the lower-depth retry applies. Each
87+
acquisition allows two lower-depth retries, and recovery allows at most 64
88+
native requests within the existing capture deadline, checks foreground
89+
ownership on every request, and returns a failure when it cannot complete a
90+
continuation. Budget exhaustion and malformed continuations use non-launch
91+
failure codes, so the route falls back without launch re-polling. At each native
92+
fragment boundary, an absent or invalid child count means unknown completeness
93+
and fails closed. Natural leaves above that boundary need no continuation
94+
evidence. Unchanged native dictionaries and child arrays are reused.
9195

9296
## Accepted-depth hints
9397

‎apple/snapshot-bridge/SnapshotBridgeCapture.m‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,15 @@
44
static NSString *const childrenKey = @"UIAccessibilitySnapshotKeyChildren";
55
static NSString *const childCountKey = @"UIAccessibilitySnapshotKeyChildrenCount";
66
static NSString *const elementKey = @"UIAccessibilitySnapshotKeyElement";
7-
static const NSUInteger maximumRequests = 32;
7+
// One request reads at most this many levels, the window the former 64-level default used, so trees
8+
// up to 65 levels keep their request sequence; deeper levels arrive through continuations. On deep
9+
// trees the AX server rejects a request that would return more than about 62 levels (measured on
10+
// iOS 26.5), so there the first request is rejected once and the halving ladder settles at 32,
11+
// which the host then sends as a depth hint.
12+
static const NSUInteger maximumRequestLevels = 65;
13+
// Sized for the daemon's default depth bound (128 levels). Each root-to-leaf path needs one
14+
// continuation per accepted-depth window, so the budget scales with the depth it must cover.
15+
static const NSUInteger maximumRequests = 64;
816

917
@interface SnapshotTreeCapture : NSObject
1018
@property(nonatomic, copy) SnapshotElementReader reader;
@@ -122,7 +130,8 @@ - (nullable NSDictionary *)materialize:(NSDictionary *)tree depth:(NSUInteger)de
122130
{
123131
SnapshotTreeCapture *capture = [SnapshotTreeCapture new];
124132
capture.reader = reader;
125-
capture.acceptedDepth = nativeLevelsHint > 0 ? MIN(maxDepth + 1, nativeLevelsHint) : maxDepth + 1;
133+
NSUInteger firstRequestLevels = nativeLevelsHint > 0 ? MIN(nativeLevelsHint, maximumRequestLevels) : maximumRequestLevels;
134+
capture.acceptedDepth = MIN(maxDepth + 1, firstRequestLevels);
126135
capture.remainingNodes = maxNodes;
127136
capture.maximumNodes = maxNodes;
128137
NSDictionary *tree = [capture read:element depth:maxDepth + 1 error:error];

‎contracts/fixtures/ios-ax-recovery-conformance.json‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@
4343
"Recovery ladder: the host halves the rejected depth at most twice per request; the runner walks 56/40/24/12 and has no rung below 12. A rejection at depth 8 recovers on the host and fails on the runner.",
4444
"Completeness evidence: the host trusts native child counts and fails closed when a boundary count is missing; the runner infers frontiers from the deepest childless level and pays one extra call to verify a natural leaf that sits at the cap (unknown-frontier-evidence, hint-rejected-then-lower).",
4545
"Delivered depth: host continuations keep the requested traversal depth and disclose deeper content as truncated; runner extension re-roots with the same native depth, so the delivered tree can exceed the requested depth and an explicit depth is not disclosed as truncated (deeper-than-requested, explicit-depth-honored).",
46-
"Budgets: the host spends one continuation per withheld parent inside a 32-request budget; the runner spends one extension call per frontier node inside an 8-call budget (host-request-budget, runner-extension-budget).",
46+
"Budgets: the host spends one continuation per withheld parent inside a 64-request budget; the runner spends one extension call per frontier node inside an 8-call budget (host-request-budget, runner-extension-budget).",
4747
"Ownership and deadlines: only the host checks the foreground owner on every native request; only the runner consults the capture deadline between rungs and extension calls (owner-change-during-continuation, deadline-before-retry).",
4848
"Hints: both producers learn from any finished lower-depth recovery, bounded trees included (bounded-recovery-still-learned); the host keys on the resolved target generation and expires by hinted-capture count, while the runner keys on bundle id + pid and expires by wall clock (probe-back-after-expiry)."
4949
],
@@ -371,7 +371,7 @@
371371
"chain": 15,
372372
"fan": {
373373
"at": 14,
374-
"count": 40,
374+
"count": 80,
375375
"chain": 1
376376
}
377377
},
@@ -382,18 +382,18 @@
382382
"host-bridge": {
383383
"outcome": "failed",
384384
"failure": "request-budget-exhausted",
385-
"requests": 32,
385+
"requests": 64,
386386
"rejected": 2,
387-
"continuations": 29
387+
"continuations": 61
388388
},
389389
"runner": {
390390
"outcome": "complete",
391391
"requests": 6,
392392
"rejected": 4,
393393
"continuations": 1,
394394
"deepestLevel": 16,
395-
"tree": "0-14,15.0-16.0,15.1-16.1,15.2-16.2,15.3-16.3,15.4-16.4,15.5-16.5,15.6-16.6,15.7-16.7,15.8-16.8,15.9-16.9,15.10-16.10,15.11-16.11,15.12-16.12,15.13-16.13,15.14-16.14,15.15-16.15,15.16-16.16,15.17-16.17,15.18-16.18,15.19-16.19,15.20-16.20,15.21-16.21,15.22-16.22,15.23-16.23,15.24-16.24,15.25-16.25,15.26-16.26,15.27-16.27,15.28-16.28,15.29-16.29,15.30-16.30,15.31-16.31,15.32-16.32,15.33-16.33,15.34-16.34,15.35-16.35,15.36-16.36,15.37-16.37,15.38-16.38,15.39-16.39",
396-
"nodes": 95
395+
"tree": "0-14,15.0-16.0,15.1-16.1,15.2-16.2,15.3-16.3,15.4-16.4,15.5-16.5,15.6-16.6,15.7-16.7,15.8-16.8,15.9-16.9,15.10-16.10,15.11-16.11,15.12-16.12,15.13-16.13,15.14-16.14,15.15-16.15,15.16-16.16,15.17-16.17,15.18-16.18,15.19-16.19,15.20-16.20,15.21-16.21,15.22-16.22,15.23-16.23,15.24-16.24,15.25-16.25,15.26-16.26,15.27-16.27,15.28-16.28,15.29-16.29,15.30-16.30,15.31-16.31,15.32-16.32,15.33-16.33,15.34-16.34,15.35-16.35,15.36-16.36,15.37-16.37,15.38-16.38,15.39-16.39,15.40-16.40,15.41-16.41,15.42-16.42,15.43-16.43,15.44-16.44,15.45-16.45,15.46-16.46,15.47-16.47,15.48-16.48,15.49-16.49,15.50-16.50,15.51-16.51,15.52-16.52,15.53-16.53,15.54-16.54,15.55-16.55,15.56-16.56,15.57-16.57,15.58-16.58,15.59-16.59,15.60-16.60,15.61-16.61,15.62-16.62,15.63-16.63,15.64-16.64,15.65-16.65,15.66-16.66,15.67-16.67,15.68-16.68,15.69-16.69,15.70-16.70,15.71-16.71,15.72-16.72,15.73-16.73,15.74-16.74,15.75-16.75,15.76-16.76,15.77-16.77,15.78-16.78,15.79-16.79",
396+
"nodes": 175
397397
}
398398
}
399399
},

‎packages/platform-apple/src/snapshot-source/adapter.test.ts‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import { createSnapshotSourceHost } from './host.ts';
1212
import { createSimulatorSnapshotSource } from './adapter.ts';
1313
import { simulatorAddressFor } from '../core/simctl.ts';
1414
import { DEPTH_HINT_PROBE_BACK_AFTER_USES } from './depth-hints.ts';
15+
import { MAXIMUM_REQUEST_LEVELS } from './limits.ts';
1516
import {
1617
encodeSnapshotBridgeFrame,
1718
SNAPSHOT_SOURCE_PROTOCOL_VERSION,
@@ -288,6 +289,53 @@ test('the Simulator AX source learns a hint only from a validated acquisition',
288289
}
289290
});
290291

292+
test('the Simulator AX source forgets a hint once a default-depth probe-back is accepted', async () => {
293+
const root = await mkdtempForTest('agent-device-snapshot-adapter-forget-');
294+
const sourceRoot = path.join(root, 'source');
295+
await (await import('@agent-device/host-kit/host-file')).ensureHostDirectory(sourceRoot);
296+
for (const name of [
297+
'SnapshotBridge.m',
298+
'SnapshotBridgeRuntime.m',
299+
'SnapshotBridgeRuntime.h',
300+
'SnapshotBridgeCapture.h',
301+
'SnapshotBridgeCapture.m',
302+
'ForegroundApplicationGeometry.h',
303+
]) {
304+
await writeFile(path.join(sourceRoot, name), 'native source');
305+
}
306+
const fixture = createAdapterHost();
307+
// Default limits: the traversal bound is deeper than one native request.
308+
const source = createSimulatorSnapshotSource({
309+
host: fixture.host,
310+
sourceRoot,
311+
cacheRoot: path.join(root, 'cache'),
312+
});
313+
const hint = deriveIosCaptureHint(createIosSnapshotRequest());
314+
const target = { ...targetForTest(), targetId: 'target-1', generation: 'generation-1' };
315+
316+
try {
317+
fixture.rejectLevelsAbove = 32;
318+
assert.equal((await source.acquire({ target, hint })).stage, 'acquired');
319+
fixture.rejectLevelsAbove = undefined;
320+
for (let use = 0; use <= DEPTH_HINT_PROBE_BACK_AFTER_USES; use++) {
321+
assert.equal((await source.acquire({ target, hint })).stage, 'acquired');
322+
}
323+
assert.equal((await source.acquire({ target, hint })).stage, 'acquired');
324+
assert.deepEqual(
325+
fixture.diagnostics
326+
.slice(-2)
327+
.map((event) => [event.requestedLevels, event.hint, event.learning]),
328+
[
329+
[129, 'probe-back', 'forgotten'],
330+
[129, 'no-hint', 'ignored'],
331+
],
332+
);
333+
} finally {
334+
await source.close();
335+
await rm(root, { recursive: true, force: true });
336+
}
337+
});
338+
291339
type HintStep = Readonly<{
292340
expire?: boolean;
293341
target?: Readonly<{ id: string; generation: string }>;
@@ -605,8 +653,12 @@ class AdapterSocket extends EventEmitter implements SnapshotSourceSocket {
605653
};
606654
this.fixture.requestedDepths.push(request.maxDepth);
607655
this.fixture.requestedHints.push(request.nativeLevelsHint);
608-
// The guest's own recovery: halve a rejected depth at most twice, like SnapshotBridgeCapture.
609-
const requestedLevels = request.nativeLevelsHint ?? request.maxDepth + 1;
656+
// The guest's own recovery: one request reads at most 65 levels, and a rejected depth halves
657+
// at most twice, like SnapshotBridgeCapture.
658+
const requestedLevels = Math.min(
659+
request.maxDepth + 1,
660+
request.nativeLevelsHint ?? MAXIMUM_REQUEST_LEVELS,
661+
);
610662
const rejectAbove = this.fixture.rejectLevelsAbove;
611663
let acceptedLevels = requestedLevels;
612664
let rejected = 0;

‎packages/platform-apple/src/snapshot-source/depth-hints.ts‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { MAXIMUM_REQUEST_LEVELS } from './limits.ts';
12
import type { SnapshotBridgeRecovery } from './protocol.ts';
23
import type { SnapshotSourceTarget } from './types.ts';
34

@@ -50,7 +51,7 @@ export class AcceptedDepthHints {
5051
if (explicitDepth) return { nativeLevels: undefined, reason: 'explicit-depth' };
5152
const entry = this.currentEntry(target);
5253
if (entry === 'unidentified') return { nativeLevels: undefined, reason: 'unidentified-target' };
53-
if (!entry || entry.nativeLevels >= requestedLevels) {
54+
if (!entry || entry.nativeLevels >= firstRequestLevels(requestedLevels)) {
5455
return { nativeLevels: undefined, reason: 'no-hint' };
5556
}
5657
if (entry.remainingUses <= 0) return { nativeLevels: undefined, reason: 'probe-back' };
@@ -65,7 +66,8 @@ export class AcceptedDepthHints {
6566
recovery: SnapshotBridgeRecovery,
6667
): DepthHintLearning {
6768
if (explicitDepth || !target.targetId) return 'ignored';
68-
const recoveredLower = recovery.rejected > 0 && recovery.acceptedLevels < requestedLevels;
69+
const firstRequest = firstRequestLevels(requestedLevels);
70+
const recoveredLower = recovery.rejected > 0 && recovery.acceptedLevels < firstRequest;
6971
if (recoveredLower) {
7072
this.remember(target.targetId, {
7173
generation: target.generation,
@@ -74,7 +76,7 @@ export class AcceptedDepthHints {
7476
});
7577
return 'learned';
7678
}
77-
const acceptedFullDepth = recovery.rejected === 0 && recovery.acceptedLevels >= requestedLevels;
79+
const acceptedFullDepth = recovery.rejected === 0 && recovery.acceptedLevels >= firstRequest;
7880
if (acceptedFullDepth) {
7981
return this.entries.delete(target.targetId) ? 'forgotten' : 'ignored';
8082
}
@@ -101,3 +103,11 @@ export class AcceptedDepthHints {
101103
this.entries.set(targetId, entry);
102104
}
103105
}
106+
107+
/**
108+
* Native levels the guest's first request asks for without a hint. Hints only change that request,
109+
* so they compare against it, not against the whole requested depth.
110+
*/
111+
function firstRequestLevels(requestedLevels: number): number {
112+
return Math.min(requestedLevels, MAXIMUM_REQUEST_LEVELS);
113+
}

‎packages/platform-apple/src/snapshot-source/fixtures/foreground-owner.m‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ - (id)userTestingSnapshotForElement:(id)element options:(NSDictionary *)options
4545
if ([captureScenario isEqual:@"runtime-budget"]) {
4646
BOOL root = ![element isKindOfClass:NSNumber.class];
4747
NSMutableArray *children = [NSMutableArray array];
48-
for (NSUInteger i = 0; i < (root ? 40 : 1); i++) [children addObject:@{
48+
for (NSUInteger i = 0; i < (root ? 80 : 1); i++) [children addObject:@{
4949
@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyChildren": @[],
5050
@"UIAccessibilitySnapshotKeyElement": @(i), @"UIAccessibilitySnapshotKeyChildrenCount": @(root ? 1 : 0)}];
5151
return @{@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyChildren": children};
@@ -153,7 +153,7 @@ int main(int argc, const char *argv[])
153153
BOOL budget = [scenario isEqual:@"request-budget"];
154154
NSUInteger depth = [scenario hasPrefix:@"api-depth-"] ? [[scenario substringFromIndex:10] integerValue] : 64;
155155
NSMutableArray *children = [NSMutableArray array];
156-
for (NSUInteger i = 0; i < (budget ? 40 : 2); i++) {
156+
for (NSUInteger i = 0; i < (budget ? 80 : 2); i++) {
157157
[children addObject:@{@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyChildren": @[],
158158
@"UIAccessibilitySnapshotKeyElement": @(i), @"UIAccessibilitySnapshotKeyChildrenCount": @(budget ? 1 : 0)}];
159159
}
@@ -164,20 +164,22 @@ int main(int argc, const char *argv[])
164164
NSDictionary *result = captureSnapshotTree(@"root", depth, 1000, 0, ^id(id element, NSUInteger levels, NSUInteger nodes, NSError **error) {
165165
requests++;
166166
if ([scenario hasPrefix:@"api-depth-"]) {
167-
require(levels == depth + 1, @"native levels must include the root exactly once");
167+
NSUInteger start = [element isKindOfClass:NSNumber.class] ? [element unsignedIntegerValue] : 0;
168+
require(levels == MIN(depth + 1 - start, 65), @"native levels must include the root exactly once, at most 65 per request");
169+
NSUInteger end = MIN(start + levels, depth + 1);
168170
NSDictionary *tree = nil;
169-
for (NSUInteger i = 0; i < levels; i++) tree = @{@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyChildrenCount": [scenario isEqual:@"api-depth-unknown"] ? [NSNull null] : @(tree ? 1 : 0), @"UIAccessibilitySnapshotKeyChildren": tree ? @[tree] : @[]};
171+
for (NSUInteger level = end; level-- > start;) tree = @{@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyElement": @(level), @"UIAccessibilitySnapshotKeyChildrenCount": [scenario isEqual:@"api-depth-unknown"] ? [NSNull null] : @(level < depth ? 1 : 0), @"UIAccessibilitySnapshotKeyChildren": tree ? @[tree] : @[]};
170172
return tree;
171173
}
172174
if ([element isEqual:@"root"]) return root;
173175
return @{@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyChildren": @[@{@"UIAccessibilitySnapshotKeyAttributes": @{}, @"UIAccessibilitySnapshotKeyChildren": @[]}]};
174176
}, &truncated, NULL, &failure);
175-
if (budget) require(!result && failure.code == 1 && requests == 32, @"request budget must fail without publishing partial content");
177+
if (budget) require(!result && failure.code == 1 && requests == 64, @"request budget must fail without publishing partial content");
176178
else if ([scenario isEqual:@"identity"]) require(result == root && requests == 1, @"healthy capture must reuse the native tree");
177179
else {
178180
NSUInteger count = 0;
179181
for (NSDictionary *node = result; node; node = [node[@"UIAccessibilitySnapshotKeyChildren"] firstObject]) count++;
180-
require(count == depth + 1 && truncated == [scenario isEqual:@"api-depth-unknown"] && requests == 1, @"every requested depth must include root plus permitted descendants");
182+
require(count == depth + 1 && truncated == [scenario isEqual:@"api-depth-unknown"] && requests == (depth + 1 > 65 ? 2 : 1), @"every requested depth must include root plus permitted descendants");
181183
}
182184
return 0;
183185
}
@@ -191,7 +193,7 @@ int main(int argc, const char *argv[])
191193
NSString *expectedCode = nil;
192194
NSUInteger expectedCaptures = 0;
193195
if ([scenario isEqual:@"runtime-budget"]) {
194-
expectedCaptures = 32;
196+
expectedCaptures = 64;
195197
expectedCode = @"continuation-budget-exhausted";
196198
} else if ([scenario hasPrefix:@"wide-"] || [scenario isEqual:@"zero-depth"]) {
197199
expectedCaptures = [scenario isEqual:@"wide-continuation"] ? 2 : 1;

‎packages/platform-apple/src/snapshot-source/limits.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,20 @@ import type { SnapshotSourceLimits } from './types.ts';
44
const FRAME_HEADER_BYTES = 4;
55
const MAXIMUM_FRAME_BYTES = 16 * 1024 * 1024;
66
const MAXIMUM_DURATION_MS = 120_000;
7+
// Mirrors the native bridge's kMaximumDepth.
78
const MAXIMUM_DEPTH = 128;
9+
// Mirrors the native bridge's maximumRequestLevels: one native request reads at most this many
10+
// levels, and deeper levels arrive through continuations.
11+
export const MAXIMUM_REQUEST_LEVELS = 65;
812
const MAXIMUM_NODES = 10_000;
913

1014
export const DEFAULT_SNAPSHOT_SOURCE_LIMITS: SnapshotSourceLimits = Object.freeze({
1115
maxRequestBytes: 64 * 1024,
1216
maxResponseBytes: 4 * 1024 * 1024,
1317
maxNodes: 5000,
14-
maxTraversalDepth: 64,
18+
// The bridge reads past the AX server's per-request depth limit through continuations, so a
19+
// regular capture asks for every level it can deliver; a lower default hides deep controls.
20+
maxTraversalDepth: MAXIMUM_DEPTH,
1521
maxDurationMs: 5_000,
1622
});
1723

0 commit comments

Comments
 (0)