Skip to content

Commit fa8cce3

Browse files
authored
refactor(ios): snapshot capture plans with a structured quality verdict (#783)
* refactor(ios): snapshot capture plans with a structured quality verdict Implements ADR 0004's explicit-strategies decision as architecture (candidates 1+2 of the snapshot pipeline review): - Snapshot backend seam: three adapters (recursive tree, query sweep, private AX) behind one captureWithBackend dispatch. Each strategy declares its chain as data (regular: tree→queries→private-ax, compact: queries→private-ax, raw: tree→private-ax) and one plan runner walks it under a 20s umbrella budget so chained recovery can never stack past the 30s main-thread watchdog. Terminal policy is per-plan: raw rethrows AX failures (diagnostics preserve errors), interactive fails closed with runnerFatal invalidation. - Single quality classifier: one sparsePayloadReason predicate (with reason codes), one collapsed-leaf detector, replacing the three divergent sparse detectors (Swift structural, daemon count==1, CLI count<=3) that each patched a different failure shape. - Structured snapshot quality verdict on the wire (state, backend, reason, reasonCode, effectiveDepth, collapsedLeafIndexes): the daemon and CLI render warnings from it instead of re-deriving degradation from node shapes; budget starvation is no longer blamed on the app's accessibility. Legacy runner messages and daemon-side detectors stay behind a verdict-absent gate for mixed-version compat. - The verdict surfaces in --json (snapshotQuality) for agents; the generic sparse CLI hint is suppressed when a verdict explains it. Threading the verdict exposed two more hand-copy field drops (captureInteractionOutcomeAwareSnapshot, serializeSnapshotResult, client response mapping) - now carried alongside warnings everywhere. Verified live: Settings healthy (tree, no warnings), Settings compact under load (recovered/private-ax/budget), production login (sparse best-effort with honest warning), collapse fixture (healthy + collapsedLeafIndexes -> @ref warning), Bluesky Home (recovered/ private-ax, 24 nodes in 2s). Full unit suite 2327 passed, fallow clean, runner builds. * fix(ios): correct recovered-snapshot viewport and private-AX scope semantics Review follow-ups on the capture-plan refactor: - The query-sweep synthetic root doubles as the daemon's viewport (find.ts prefers on-screen matches inside nodes[0].rect), but it was built from candidate bounds, so off-screen controls below the screen could inflate it and win duplicate-label resolution. The root now uses the real finite viewport, falling back to candidate bounds only when viewport capture failed. - The private-AX backend applied --scope as a per-node text filter, hiding the matched container's children — diverging from regular snapshot scope semantics and contradicting the depth-cap hint that recommends scoped re-runs. Scope now selects the matched subtree: descendants inherit the match and only the normal option filters apply to them (in-bundle test covers a non-matching descendant). Verified live on Bluesky Home: scope homeScreen returns the 52-node subtree including non-matching descendants; compact root rect equals the screen (0,0,402,874). * fix(ios): fail closed on interactive AX failure, stamp fatal verdict, validate parser Three review findings on the capture-plan terminal path: - P1: the fail-closed guard required `best == nil`, but the query-sweep tier always returns a synthetic-root sparse payload that sets `best` — so an interactive recursive-tree AX serialization failure that no backend recovered returned a sparse snapshot instead of invalidating the cached target. Reaching the terminal already means no backend produced a usable tree, so the sparse `best` must not suppress the fail-closed path. Extracted the decision into a pure, unit-tested `resolveSnapshotPlanTerminal` (closes the terminal-ordering testability gap the architecture review flagged). - P2: `snapshotAccessibilityUnavailable` returned a payload with no `snapshotQuality`, leaving one planned sparse result on the legacy-message path. It now carries a sparse/ax-rejected verdict like every other planned snapshot, so downstream sparse handling keys off the verdict. - P2: `readSnapshotQualityVerdict` cast any string state/backend into the union, so a malformed object suppressed the legacy node-shape detectors. State and backend are now validated against their unions (unknown → verdict-absent → legacy detectors run); an unknown reasonCode is dropped rather than rejecting the whole verdict, so a forward-version runner still yields a usable verdict. Unit-covered: Swift resolveSnapshotPlanTerminal matrix + fatal-verdict assertion; TS parser accept/reject/forward-compat. Full suite 249 files / 2449 tests, fallow, lint, runner build green.
1 parent 3b94b36 commit fa8cce3

18 files changed

Lines changed: 819 additions & 444 deletions

‎CONTEXT.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
- Daemon command registry: daemon-side source of truth for command route ownership and request-policy traits, including admission exemptions, session locking, selector validation, replay-scoped actions, recording invalidation, Android dialog guards, and request provider device resolution.
1818
- Runner command traits: the iOS XCTest runner's per-command-type classification across three independent axes — interaction (gates the foreground-guard and stabilization preflight), read-only (gates the session-invalidating retry; the alert command is read-only only for its `get` action), and runner-lifecycle (skips the app-activation preflight). One source of truth keyed by command type, distinct from the public command surface and daemon command registry.
1919
- Coordinate-first resolved element activation: iOS/macOS runner interaction pattern where a selector or text query resolves the semantic `XCUIElement`, then activation uses the element's resolved center coordinate when a frame is available. This keeps target selection semantic while avoiding `XCUIElement.tap()` post-action element re-resolution after normal navigation. tvOS remains focus/remote-driven.
20+
- Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree, query sweep, private AX) run by one plan runner under a shared wall-clock budget; recovery ordering is declared data, never a per-call-site branch.
21+
- Snapshot quality verdict: structured outcome (state, backend, reason code, effective depth, collapsed leaves) computed once by the plan runner and shipped with every planned snapshot payload; the daemon and CLI render it instead of re-deriving degradation from node shapes.
2022
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect. The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the next command reacquire the app through normal activation.
2123

2224
## Testing Principles

‎docs/adr/0004-ios-snapshot-backend-strategy.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,9 @@
22

33
## Status
44

5-
Accepted
5+
Accepted — implemented by the snapshot capture plan runner (RunnerTests+SnapshotCapturePlan.swift):
6+
each strategy declares its backend chain, and a structured snapshot quality verdict makes
7+
degraded or recovered output observable end to end.
68

79
## Context
810

‎ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift‎

Lines changed: 66 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,10 @@ extension RunnerTests {
1010
/// apps where the AX surface is genuinely unavailable.
1111
static let privateAXSnapshotDepthLadder = [56, 40, 24, 12]
1212

13-
func privateAXSnapshotFallback(
13+
func privateAXSnapshotCapture(
1414
app: XCUIApplication,
15-
options: SnapshotOptions,
16-
reason: String
17-
) -> DataPayload? {
15+
options: SnapshotOptions
16+
) -> SnapshotBackendCapture? {
1817
#if os(iOS) && targetEnvironment(simulator)
1918
let requestedDepth = options.depth ?? 64
2019
var attemptDepths = [requestedDepth]
@@ -66,29 +65,24 @@ extension RunnerTests {
6665
options: options,
6766
viewport: viewport,
6867
depth: 0,
69-
parentIndex: nil
68+
parentIndex: nil,
69+
insideMatchedScope: false
7070
)
7171
if nodes.count <= 1 {
7272
NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_SPARSE=%ld", nodes.count)
7373
return nil
7474
}
7575

7676
let depthLimited = effectiveDepth < requestedDepth
77-
let truncated = (response["truncated"] as? Bool) == true || depthLimited
78-
var message =
79-
"Recovered this snapshot with the fallback accessibility backend after \(reason). This usually means the app publishes an unhealthy accessibility tree (too large or deep to serialize, or containers that hide their children) — fixing the app's accessibility is the real cure. The fallback is simulator-only and may expose a partial tree; treat screenshot as visual truth when this warning appears."
80-
if depthLimited {
81-
message +=
82-
" The accessibility server rejected deeper requests; this tree is capped at depth \(effectiveDepth) — re-run with --depth \(effectiveDepth) --scope <container> to inspect deeper content."
83-
}
8477
NSLog(
85-
"AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_USED reason=%@ nodes=%ld depth=%ld truncated=%@",
86-
reason,
78+
"AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_USED nodes=%ld depth=%ld",
8779
nodes.count,
88-
effectiveDepth,
89-
truncated ? "true" : "false"
80+
effectiveDepth
81+
)
82+
return SnapshotBackendCapture(
83+
payload: DataPayload(nodes: nodes, truncated: (response["truncated"] as? Bool) == true),
84+
effectiveDepth: depthLimited ? effectiveDepth : nil
9085
)
91-
return DataPayload(message: message, nodes: nodes, truncated: truncated)
9286
#else
9387
return nil
9488
#endif
@@ -100,7 +94,8 @@ extension RunnerTests {
10094
options: SnapshotOptions,
10195
viewport: CGRect,
10296
depth: Int,
103-
parentIndex: Int?
97+
parentIndex: Int?,
98+
insideMatchedScope: Bool
10499
) {
105100
if let limit = options.depth, depth > limit { return }
106101

@@ -115,14 +110,26 @@ extension RunnerTests {
115110
let hasContent = !label.isEmpty || !identifier.isEmpty || !value.isEmpty
116111
let isRoot = parentIndex == nil
117112

113+
// Scope selects a subtree, matching regular snapshot semantics: once a node matches,
114+
// every descendant is inside scope and only the normal option filters apply to it.
115+
let scope = options.scope?.trimmingCharacters(in: .whitespacesAndNewlines)
116+
let scopeActive = (scope?.isEmpty == false)
117+
let matchesScope: Bool
118+
if scopeActive, let scope {
119+
let haystack = [label, identifier, value].joined(separator: "\n")
120+
matchesScope = haystack.localizedCaseInsensitiveContains(scope)
121+
} else {
122+
matchesScope = false
123+
}
124+
let nowInsideScope = insideMatchedScope || matchesScope
125+
118126
let include: Bool
119127
if isRoot {
120128
include = true
129+
} else if scopeActive && !nowInsideScope {
130+
include = false
121131
} else if options.interactiveOnly && !visible {
122132
include = false
123-
} else if let scope = options.scope?.trimmingCharacters(in: .whitespacesAndNewlines), !scope.isEmpty {
124-
let haystack = [label, identifier, value].joined(separator: "\n")
125-
include = haystack.localizedCaseInsensitiveContains(scope)
126133
} else if options.compact {
127134
include = hasContent || privateAXLikelyInteractive(rawElementType: rawType)
128135
} else {
@@ -164,7 +171,8 @@ extension RunnerTests {
164171
options: options,
165172
viewport: viewport,
166173
depth: depth + 1,
167-
parentIndex: currentIndex
174+
parentIndex: currentIndex,
175+
insideMatchedScope: nowInsideScope
168176
)
169177
}
170178
}
@@ -225,3 +233,39 @@ extension RunnerTests {
225233
return nil
226234
}
227235
}
236+
237+
// MARK: - In-bundle unit tests
238+
239+
extension RunnerTests {
240+
func testPrivateAXScopeSelectsSubtreeNotMatchingLabels() {
241+
let tree: [String: Any] = [
242+
"type": 1, "label": "App",
243+
"children": [
244+
[
245+
"type": 9, "identifier": "homeScreen",
246+
"children": [
247+
["type": 48, "label": "Post body without the scope text", "children": []]
248+
],
249+
],
250+
["type": 9, "label": "unrelated sibling", "children": []],
251+
],
252+
]
253+
var nodes: [SnapshotNode] = []
254+
appendPrivateAXNode(
255+
tree,
256+
to: &nodes,
257+
options: SnapshotOptions(
258+
interactiveOnly: false, compact: false, depth: nil, scope: "homeScreen", raw: false),
259+
viewport: .infinite,
260+
depth: 0,
261+
parentIndex: nil,
262+
insideMatchedScope: false
263+
)
264+
265+
let labels = nodes.compactMap { $0.label ?? $0.identifier }
266+
XCTAssertTrue(labels.contains("homeScreen"))
267+
// Descendants of the matched scope are included even when they do not contain the text.
268+
XCTAssertTrue(labels.contains("Post body without the scope text"))
269+
XCTAssertFalse(labels.contains("unrelated sibling"))
270+
}
271+
}

‎ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,7 @@ struct DataPayload: Codable {
205205
let items: [String]?
206206
let nodes: [SnapshotNode]?
207207
let truncated: Bool?
208+
let snapshotQuality: SnapshotQuality?
208209
let gestureStartUptimeMs: Double?
209210
let gestureEndUptimeMs: Double?
210211
let x: Double?
@@ -242,6 +243,7 @@ struct DataPayload: Codable {
242243
items: [String]? = nil,
243244
nodes: [SnapshotNode]? = nil,
244245
truncated: Bool? = nil,
246+
snapshotQuality: SnapshotQuality? = nil,
245247
gestureStartUptimeMs: Double? = nil,
246248
gestureEndUptimeMs: Double? = nil,
247249
x: Double? = nil,
@@ -278,6 +280,7 @@ struct DataPayload: Codable {
278280
self.items = items
279281
self.nodes = nodes
280282
self.truncated = truncated
283+
self.snapshotQuality = snapshotQuality
281284
self.gestureStartUptimeMs = gestureStartUptimeMs
282285
self.gestureEndUptimeMs = gestureEndUptimeMs
283286
self.x = x

0 commit comments

Comments
 (0)