Skip to content

Commit 4a3ed1e

Browse files
authored
feat(ios): extend depth-capped private-AX captures via element-rooted requests (#1627)
* feat(ios): extend depth-capped private-AX captures via element-rooted requests The AX server's depth limit is per-request (kAXErrorIllegalArgument above a tree-size-dependent value), so a capture capped at depth 56 on Bluesky-class React Native trees returned chains of unlabeled [other] containers and hid every actionable control below the cap — agents fell back to screenshot-and-coordinate guessing. After a capped serialization, the bridge now re-issues the same snapshot request rooted at each deepest-level childless node's live accessibility element, splicing the returned children in and chaining further while capped subtrees remain — bounded by a call limit, the shared node budget, and the capture-plan deadline. The AX server counts node levels, not edges: a maxDepth=56 request emits nodes to depth 55, so the frontier is the deepest observed level, kept only when it sits at the cap; a tree that ends naturally above the cap yields no frontiers and costs nothing. A capture whose frontier extension drained every capped node no longer reports itself depth-limited — the re-run hint it used to trigger could not add anything. Explicit --depth requests stay exact captures with no extension. Live on the seeded Bluesky bench feed: 8 extension calls (~106ms each) turn the 117-node all-[other] tree into a 182-node tree carrying post text, testID links, and every feed control (Reply/Repost/Like/options); steady-state capture 0.52s -> 1.37s. Observation-only captures paying the extension needlessly is #1626. * fix: count missed frontiers and tighten deep-extension shape (review) The completeness verdict inverted in the failure paths: a frontier whose live element vanished or whose re-rooted request failed was silently dropped, so an all-miss extension reported pendingFrontiers=0 and the capture presented as complete while whole subtrees were missing. Missed frontiers are now counted, logged, and keep the depth-limited verdict — the pure decision lives in privateAXDepthLimited with in-bundle tests. The truncation hint no longer advertises --depth (an explicit --depth capture disables extension, so following it returned strictly less than the capture that produced the hint); the honest remedy is a plain re-run with a fresh extension budget. Shape: candidates collected only at the cap boundary and not at all when extension is disabled (exact --depth captures pay zero bookkeeping); the zero-caller 3-arg overload is gone; the response shape's keys are shared constants; the unsupported-selector diagnostic is restored; the exact-depth policy is hoisted to one named local. * test: execute the deep-extension miss-path contract in CI (review) The depth-limited regression compiled but never ran (absent from ios.yml's -only-testing list), and the pure Swift consumer test was vacuous against the Objective-C producer — a literal missedFrontiers proves nothing about the increments. RunnerAXSnapshotFrontier and extendSnapshotFrontiers move to the header as the executed contract seam, and testDeepExtensionCountsMissedFrontiers drives both real miss paths with fabricated snapshots: a nil accessibilityElement (explicit nil property — bare NSObject resolves the key through a UIKit category and takes the call path instead) counts missed without consuming a call; a resolving element whose client cannot serve the re-rooted request consumes a call AND counts missed. Both tests join the executed ios.yml list. Red-before verified on-simulator: with both increments stripped the producer test fails at the counter assertion (0 != 2); restored, both tests pass.
1 parent 46abd0f commit 4a3ed1e

5 files changed

Lines changed: 472 additions & 104 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ jobs:
106106
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrain \
107107
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrainForSnapshotRaw \
108108
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBounds \
109+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \
110+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \
109111
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledNeedsEnoughSamples \
110112
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledTrueWhenWindowMatches \
111113
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledFalseOnMidWindowMismatch \

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerAXSnapshotBridge.h‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,54 @@
33

44
NS_ASSUME_NONNULL_BEGIN
55

6+
/// Keys of the `deepExtension` dictionary in the snapshot response, shared
7+
/// with the Swift consumer so the response shape has one spelling.
8+
FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionKey;
9+
FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionCallsKey;
10+
FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionNodesAddedKey;
11+
FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionPendingKey;
12+
FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionMissedKey;
13+
14+
/// A depth-capped childless node awaiting an element-rooted follow-up request.
15+
/// Public so the runner unit bundle can drive the extension's miss paths with
16+
/// fabricated snapshots — the executed contract that missed frontiers are
17+
/// counted, never silently dropped.
18+
@interface RunnerAXSnapshotFrontier : NSObject
19+
@property(nonatomic, strong, nullable) id snapshot;
20+
@property(nonatomic, strong, nullable) NSMutableDictionary *node;
21+
@property(nonatomic, assign) NSInteger depth;
22+
@end
23+
624
@interface RunnerAXSnapshotBridge : NSObject
725

26+
/// Captures the app's accessibility tree, then chains element-rooted follow-up
27+
/// requests from depth-capped frontier nodes (the AX depth limit is
28+
/// per-request, so re-rooting reaches content the app-rooted request could
29+
/// not). The response gains a `deepExtension` {calls, nodesAdded,
30+
/// pendingFrontiers, missedFrontiers} dictionary when any frontier existed and
31+
/// the call limit allowed extension. Pass 0 to disable extension entirely.
832
+ (NSDictionary<NSString *, id> *)snapshotTreeForApplication:(XCUIApplication *)application
933
maxDepth:(NSInteger)maxDepth
10-
maxNodes:(NSInteger)maxNodes;
34+
maxNodes:(NSInteger)maxNodes
35+
deepExtensionCallLimit:(NSInteger)deepExtensionCallLimit
36+
deadline:(nullable NSDate *)deadline;
1137

1238
+ (NSInteger)processIdentifierForApplication:(XCUIApplication *)application;
1339

40+
/// The frontier-extension loop, exposed for the runner unit bundle: a frontier
41+
/// whose element vanished or whose re-rooted request fails must count as
42+
/// missed (see the deepExtension keys above) — an all-miss extension reporting
43+
/// itself drained would present a capped capture as complete.
44+
+ (nullable NSDictionary *)extendSnapshotFrontiers:(NSMutableArray<RunnerAXSnapshotFrontier *> *)frontiers
45+
axClient:(id)axClient
46+
attributes:(NSArray *)attributes
47+
maxDepth:(NSInteger)maxDepth
48+
maxNodes:(NSInteger)maxNodes
49+
nodeCount:(NSInteger *)nodeCount
50+
truncated:(BOOL *)truncated
51+
callsAllowed:(NSInteger)callsAllowed
52+
deadline:(nullable NSDate *)deadline;
53+
1454
@end
1555

1656
NS_ASSUME_NONNULL_END

0 commit comments

Comments
 (0)