Skip to content

Commit b121bcd

Browse files
committed
feat(ios): unify snapshot eligibility
Make iOS regular snapshot eligibility one backend-neutral presentation rule. Acquire tree nodes conservatively, preserve interactive scroll containers, normalize surviving hierarchy, and keep raw membership plus daemon publication policy unchanged. Part of #1797. - iOS and macOS unit-enabled runner builds - 2 focused XCTest cases - 3 production-path publication tests - live Settings snapshots: 73 regular nodes and 167 raw nodes, both healthy tree captures
1 parent a70cdee commit b121bcd

8 files changed

Lines changed: 335 additions & 74 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@ jobs:
118118
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrainForSnapshotRaw \
119119
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBounds \
120120
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationPreservesCurrentWireShape \
121+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsBackendNeutralEligibility \
121122
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \
122123
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \
123124
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Unreleased
44

5+
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
56
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.
67
- New `hover <x y|@ref|selector>` command for `--platform web` (#1783). It moves the pointer over the target without pressing, so hover-gated UI — a message row's `...` toolbar, a menu that opens on pointer enter — becomes reachable through agent-device the way it already was through the underlying `agent-browser` backend (`mouse move`). It is a member of the targeted-touch family: same `@ref`/selector/coordinate targeting, occlusion and off-screen guards, and `--settle` (the settled diff carries the revealed controls with fresh refs, e.g. `+ @e4 [button] "Delete"`), but no `--verify`, since hover reveals rather than activates. `hover @ref` publishes as a portable selector line in recorded scripts, and the Node client exposes `interactions.hover`. Hover is a pointer state that touch platforms do not have, so `capabilities` advertises it on web only and iOS/Android/Linux reject it during admission with `UNSUPPORTED_OPERATION` and a hint naming `--platform web`; `longpress` remains the mobile hold-gesture verb.
78
- Internal: session recording is now derived from the script-publication lifecycle instead of being stored beside it. `SessionState.recordSession` is removed; `isRecordingPublication` answers the question from the aggregate (ordinary authoring records only while ARMED, a repair transaction records for its whole lifetime). The stored flag was a second source of truth that handler surfaces set directly, which is how #1533's aborted-recording drift arose — no surface can now arm recording without moving the lifecycle that authorizes it, and the script writer's publication gate is answered entirely by the aggregate. Behavior-preserving: the derivation reproduces what the flag held at every transition.

‎CONTEXT.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,11 @@ task touches:
224224
- Presentation options: the source-of-truth iOS snapshot request policy accepted by
225225
`SnapshotPresentation`. During the behavior-preserving #1797 migration acquisition still reads
226226
these options to reproduce current backend-specific decisions.
227+
- Snapshot eligibility: runner-presentation membership for the regular projection. A root carrier
228+
survives for viewport geometry; every other node needs an interactive accessibility type or a
229+
non-empty label, identifier, or value. Hittability is not eligibility, and raw membership is
230+
exempt. Removing a wrapper reparents surviving descendants and normalizes indexes and depths.
231+
Daemon compaction separately owns publication membership and its declared suppressions.
227232
- Presented node: wire-facing iOS snapshot value constructed only by `SnapshotPresentation`; response
228233
payload assembly accepts this type rather than backend-owned raw values.
229234
- Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree,

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -240,7 +240,7 @@ extension RunnerTests {
240240
hints: &hiddenContentHintsByNodeIndex
241241
)
242242
}
243-
let include = shouldInclude(
243+
let include = shouldAcquireSnapshotNode(
244244
snapshot: snapshot,
245245
label: evaluation.label,
246246
identifier: evaluation.identifier,
@@ -429,7 +429,7 @@ extension RunnerTests {
429429
if let limit = options.depth, depth > limit { return }
430430

431431
let evaluation = evaluateSnapshot(snapshot, in: context)
432-
let include = shouldInclude(
432+
let include = shouldAcquireSnapshotNode(
433433
snapshot: snapshot,
434434
label: evaluation.label,
435435
identifier: evaluation.identifier,
@@ -870,7 +870,9 @@ extension RunnerTests {
870870

871871
// MARK: - Snapshot Filtering
872872

873-
private func shouldInclude(
873+
/// Conservative acquisition gate. Regular capture keeps every node its visibility policy can
874+
/// supply; `SnapshotPresentation` alone decides eligibility after the backend returns.
875+
private func shouldAcquireSnapshotNode(
874876
snapshot: XCUIElementSnapshot,
875877
label: String,
876878
identifier: String,
@@ -887,22 +889,22 @@ extension RunnerTests {
887889
return false
888890
}
889891
#endif
890-
if options.interactiveOnly {
891-
if isScrollableContainer(snapshot, visible: visible) { return true }
892+
if regularSnapshot {
892893
#if os(macOS)
893-
if !visible && type != .application {
894+
if options.interactiveOnly && !visible && type != .application {
894895
return false
895896
}
896897
#endif
898+
if type == .application || type == .window { return true }
899+
return visible
900+
}
901+
if options.interactiveOnly {
902+
if isScrollableContainer(snapshot, visible: visible) { return true }
897903
if interactiveTypes.contains(type) { return true }
898904
if hittable && type != .other { return true }
899905
if hasContent { return true }
900906
return false
901907
}
902-
if regularSnapshot {
903-
if type == .application || type == .window { return true }
904-
return visible
905-
}
906908
return true
907909
}
908910

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift‎

Lines changed: 84 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,8 @@ import Foundation
22

33
/// Backend-owned snapshot output before it crosses the presentation seam.
44
///
5-
/// Step 1 deliberately carries the fields computed by today's backend-specific implementations.
6-
/// Later #1797 migration steps move those decisions behind `SnapshotPresentation` without changing
7-
/// either the acquisition interface or the wire-facing `PresentedNode` type.
5+
/// The incremental #1797 migration still carries derived fields that later semantic layers move
6+
/// behind `SnapshotPresentation`. Eligibility is no longer one of those backend-owned decisions.
87
struct RawAXNode {
98
let index: Int
109
let type: String
@@ -57,7 +56,16 @@ struct PresentedNode: Codable {
5756
let actions: [String]?
5857

5958
fileprivate init(presenting raw: RawAXNode) {
60-
index = raw.index
59+
self.init(
60+
presenting: raw,
61+
index: raw.index,
62+
depth: raw.depth,
63+
parentIndex: raw.parentIndex
64+
)
65+
}
66+
67+
fileprivate init(presenting raw: RawAXNode, index: Int, depth: Int, parentIndex: Int?) {
68+
self.index = index
6169
type = raw.type
6270
label = raw.label
6371
identifier = raw.identifier
@@ -67,26 +75,49 @@ struct PresentedNode: Codable {
6775
focused = raw.focused
6876
selected = raw.selected
6977
hittable = raw.hittable
70-
depth = raw.depth
71-
parentIndex = raw.parentIndex
78+
self.depth = depth
79+
self.parentIndex = parentIndex
7280
hiddenContentAbove = raw.hiddenContentAbove
7381
hiddenContentBelow = raw.hiddenContentBelow
7482
actions = raw.actions
7583
}
7684
}
7785

7886
enum SnapshotPresentation {
87+
private static let eligibleInteractiveTypes: Set<String> = [
88+
"Button",
89+
"Cell",
90+
"CheckBox",
91+
"CollectionView",
92+
"Link",
93+
"MenuItem",
94+
"Picker",
95+
"SearchField",
96+
"SecureTextField",
97+
"SegmentedControl",
98+
"Slider",
99+
"ScrollView",
100+
"Stepper",
101+
"Switch",
102+
"TabBar",
103+
"Table",
104+
"TextField",
105+
"TextView",
106+
"WebView",
107+
]
108+
79109
/// The capture plan's single presentation route for every snapshot backend.
80110
///
81-
/// Step 2 preserves each backend's current decisions. `options` is deliberately part of the
82-
/// stable interface now; later #1797 steps move those decisions behind this seam.
111+
/// Eligibility is the first intentional step-3 semantic delta: regular presentation keeps a
112+
/// root carrier plus nodes with an interactive type or non-empty semantic content. Raw snapshots
113+
/// retain their acquired membership until the dedicated raw-projection migration lands.
83114
static func present(
84115
_ acquisition: SnapshotAcquisition,
85-
options _: PresentationOptions
116+
options: PresentationOptions
86117
) -> SnapshotBackendCapture {
87118
SnapshotBackendCapture(
88119
payload: DataPayload(
89-
nodes: acquisition.nodes.map(PresentedNode.init(presenting:)),
120+
nodes: presentedNodes(from: acquisition.nodes, options: options),
90121
truncated: acquisition.truncated
91122
),
92123
effectiveDepth: acquisition.effectiveDepth,
@@ -99,65 +130,54 @@ enum SnapshotPresentation {
99130
static func singleElementRead(_ node: RawAXNode) -> PresentedNode {
100131
PresentedNode(presenting: node)
101132
}
102-
}
103133

104-
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
105-
import XCTest
134+
private static func presentedNodes(
135+
from rawNodes: [RawAXNode],
136+
options: PresentationOptions
137+
) -> [PresentedNode] {
138+
if options.raw {
139+
return rawNodes.map(PresentedNode.init(presenting:))
140+
}
106141

107-
extension RunnerTests {
108-
func testSnapshotPresentationPreservesCurrentWireShape() throws {
109-
// Non-vacuity: setting PresentedNode.label to nil made this test execute once and fail on the
110-
// missing `"label":"Continue"` field before the production mapping was restored.
111-
let raw = RawAXNode(
112-
index: 3,
113-
type: "Button",
114-
label: "Continue",
115-
identifier: "continue-button",
116-
value: "Ready",
117-
rect: SnapshotRect(x: 10, y: 20, width: 100, height: 44),
118-
enabled: true,
119-
focused: true,
120-
selected: true,
121-
hittable: true,
122-
depth: 2,
123-
parentIndex: 1,
124-
hiddenContentAbove: true,
125-
hiddenContentBelow: true,
126-
actions: ["Open menu"]
127-
)
128-
let encoder = JSONEncoder()
129-
encoder.outputFormatting = [.sortedKeys]
142+
var nodes: [PresentedNode] = []
143+
var nearestPresentedNodeByRawIndex: [Int: (index: Int, depth: Int)] = [:]
144+
for raw in rawNodes {
145+
let presentedParent = raw.parentIndex.flatMap {
146+
nearestPresentedNodeByRawIndex[$0]
147+
}
148+
guard isEligibleForRegularPresentation(raw) else {
149+
if let presentedParent {
150+
nearestPresentedNodeByRawIndex[raw.index] = presentedParent
151+
}
152+
continue
153+
}
130154

131-
let capture = SnapshotPresentation.present(
132-
SnapshotAcquisition(
133-
nodes: [raw],
134-
truncated: true,
135-
effectiveDepth: 4,
136-
customActions: SnapshotCustomActionCoverage(
137-
read: 1,
138-
candidates: 2,
139-
truncated: 0,
140-
blocked: false
155+
let presentedIndex = nodes.count
156+
let presentedDepth = presentedParent.map { $0.depth + 1 } ?? 0
157+
nearestPresentedNodeByRawIndex[raw.index] = (presentedIndex, presentedDepth)
158+
nodes.append(
159+
PresentedNode(
160+
presenting: raw,
161+
index: presentedIndex,
162+
depth: presentedDepth,
163+
parentIndex: presentedParent?.index
141164
)
142-
),
143-
options: PresentationOptions(
144-
interactiveOnly: true,
145-
depth: nil,
146-
scope: nil,
147-
raw: false
148165
)
149-
)
150-
let nodes = try XCTUnwrap(capture.payload.nodes)
151-
let encoded = try encoder.encode(nodes)
166+
}
167+
return nodes
168+
}
152169

153-
XCTAssertEqual(
154-
String(decoding: encoded, as: UTF8.self),
155-
#"[{"actions":["Open menu"],"depth":2,"enabled":true,"focused":true,"hiddenContentAbove":true,"hiddenContentBelow":true,"hittable":true,"identifier":"continue-button","index":3,"label":"Continue","parentIndex":1,"rect":{"height":44,"width":100,"x":10,"y":20},"selected":true,"type":"Button","value":"Ready"}]"#
156-
)
157-
XCTAssertEqual(capture.payload.truncated, true)
158-
XCTAssertEqual(capture.effectiveDepth, 4)
159-
XCTAssertEqual(capture.customActions?.read, 1)
160-
XCTAssertEqual(capture.customActions?.candidates, 2)
170+
private static func isEligibleForRegularPresentation(_ node: RawAXNode) -> Bool {
171+
// The top-level carrier owns viewport geometry and must survive even for query-sweep's
172+
// deliberately unlabeled synthetic Application node.
173+
if node.parentIndex == nil { return true }
174+
return eligibleInteractiveTypes.contains(node.type) || hasSemanticContent(node)
175+
}
176+
177+
private static func hasSemanticContent(_ node: RawAXNode) -> Bool {
178+
[node.label, node.identifier, node.value].contains { value in
179+
guard let value else { return false }
180+
return !value.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
181+
}
161182
}
162183
}
163-
#endif

0 commit comments

Comments
 (0)