Skip to content

Commit 6ea9f81

Browse files
thymikeeclaude
andauthored
perf(ios): admit tap text-input probes from fresh target evidence (#3881)
* perf(ios): admit tap text-input probes from fresh target evidence * fix(ios): retain tap input geometry before snapshot presentation * fix(ios): preserve conservative tap probe admission * test(ios): pin a tap request carrying text-input evidence The macOS host lane requires every runner request field to have a production request in runner-requests.json; the new coordinateTapTextInputEvidence field had none. Co-Authored-By: Claude Opus 5.5 <[email protected]> * perf(capture-kit): import private evidence through its own subpath The iOS snapshot presenter imported the text-input geometry helpers through the contracts capture facade, which made it evaluate seven more modules on import. A narrow @agent-device/contracts/snapshot-private-evidence export leaves one: the private-evidence module itself. Co-Authored-By: Claude Opus 5.5 <[email protected]> * chore(gates): approve private evidence in the iOS presenter closure Approved by the maintainer for #3881: the presenter now evaluates the capture-private evidence module to attach text-input geometry. Records the new @agent-device/contracts/snapshot-private-evidence subpath in the contracts export snapshot. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(contracts): import text-input geometry only through its subpath Drop the capture facade re-export so capture-private evidence has one import spelling. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): draw tap text-input evidence only from the point's observation Target resolution now returns the observation that measured the dispatch point (pointObservation), absent when the #1542 rescue replaced the captured rect with a live one. tap() forwards it through BackendTapOptions and the daemon builds exclusion evidence from it alone, while it is still the session's current, non-outdated observation. The selector/ref branching and the ref-frame checks in the evidence closure collapse into that one check. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(contracts): keep the capture facade exhaustive over private evidence The facade-surface gate requires a facade that re-exports from snapshot-private-evidence to list every symbol. Importers keep the subpath spelling. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(daemon): refuse unconfirmed off-screen selectors in the touch fake The fake now dispatches an off-screen selector only at a live rect the rescue confirms, as the command family does. The ref-frame test gives the newer observation a different element at the ref, so production would also keep the frame node. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(contracts): keep the facade spelling for text-input geometry consumers Fallow requires the facade re-export to have consumers while the facade-surface gate requires it to stay exhaustive. Moving the helpers to their own module would change the approved closure-budget row, so consumers keep the facade and only the presenter keeps the subpath. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(interaction): read the tap's point observation from preAction membership The resolved target carried the same tree twice: `preAction` and `pointObservation`. tapCommand now forwards `preAction` as the point observation only when the dispatched node is one of its nodes. The #1542 rescue returns a patched copy, so a rescued point has no observation, and any future node-replacing stage fails closed the same way. The diagnostic `source` label leaves the wire, so the daemon no longer needs the `tapTarget` runtime input or its `selector-capture` default, and `target` returns to `iosTapCorroboration`. The runner log keeps `reason=`. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent f552d87 commit 6ea9f81

32 files changed

Lines changed: 1504 additions & 24 deletions

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandDispatch.swift‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,11 +376,15 @@ extension RunnerTests {
376376
routeToSpringboard: routeToSpringboard
377377
)
378378
let activeApp: XCUIApplication
379+
let activeAppBundleId: String?
379380
switch preparation {
380381
case .response(let response):
381382
return response
382383
case .context(let context):
383384
activeApp = context.app
385+
activeAppBundleId = routeToSpringboard
386+
? Self.springboardBundleId
387+
: context.systemSurface?.bundleId ?? mainOwned.bundleId
384388
}
385389

386390
switch command.command {
@@ -483,6 +487,7 @@ extension RunnerTests {
483487
return try executeOnMainPrepared(
484488
command: command,
485489
activeApp: activeApp,
490+
activeAppBundleId: activeAppBundleId,
486491
alertDeadline: alertDeadline
487492
)
488493
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ extension RunnerTests {
66
func executeOnMainPrepared(
77
command: Command,
88
activeApp: XCUIApplication,
9+
activeAppBundleId: String? = nil,
910
alertDeadline: Date? = nil
1011
) throws -> Response {
1112
var activeApp = activeApp
@@ -147,17 +148,22 @@ extension RunnerTests {
147148
bundleId: mainOwned.bundleId
148149
)
149150
let xCTestTextInputProbeSkipped = !shouldProbeCoordinateTapTextInput(
150-
xCTestChannelPenalized: xCTestChannelPenalized
151+
xCTestChannelPenalized: xCTestChannelPenalized,
152+
evidence: command.coordinateTapTextInputEvidence,
153+
point: CGPoint(x: x, y: y),
154+
targetBundleId: activeAppBundleId
151155
)
152156
let textInput: XCUIElement?
153157
if !xCTestTextInputProbeSkipped {
154158
textInput = coordinateTapTextInputAt(app: activeApp, x: x, y: y)
155159
} else {
156-
// A process-scoped tap cannot authorize later typing without concrete element identity.
157160
textInput = nil
161+
// Clear before dispatch so a throwing gesture cannot retain the preceding field's witness.
162+
clearRememberedTextEntryTap()
158163
NSLog(
159-
"AGENT_DEVICE_RUNNER_COORDINATE_TAP_TEXT_INPUT_PROBE_SKIPPED bundle=%@",
160-
mainOwned.bundleId ?? ""
164+
"AGENT_DEVICE_RUNNER_COORDINATE_TAP_TEXT_INPUT_PROBE_SKIPPED bundle=%@ reason=%@",
165+
mainOwned.bundleId ?? "",
166+
xCTestChannelPenalized ? "penalized-xctest-channel" : "evidence"
161167
)
162168
}
163169
var fallback: GestureFallback?

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,12 @@ extension Command {
213213
}
214214
}
215215

216+
struct CoordinateTapTextInputEvidence: Codable {
217+
let targetBundleId: String
218+
let point: RunnerGesturePoint
219+
let textInputRects: [RunnerGestureViewport]
220+
}
221+
216222
struct Command: Codable {
217223
let command: CommandType
218224
let commandId: String?
@@ -251,6 +257,7 @@ struct Command: Codable {
251257
let fullscreen: Bool?
252258
let inlineScreenshot: Bool?
253259
let synthesized: Bool?
260+
let coordinateTapTextInputEvidence: CoordinateTapTextInputEvidence?
254261
let steps: [SequenceStep]?
255262
}
256263

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedGesturePolicy.swift‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import XCTest
2+
import AgentDeviceSnapshotPresentation
23

34
// Runner-local policy for AX-free/private synthesized iOS gestures.
45
//
@@ -113,8 +114,23 @@ func synthesizedGesturePolicy(_ kind: SynthesizedGesturePolicyKind) -> Synthesiz
113114
}
114115
}
115116

116-
func shouldProbeCoordinateTapTextInput(xCTestChannelPenalized: Bool) -> Bool {
117-
!xCTestChannelPenalized
117+
func shouldProbeCoordinateTapTextInput(
118+
xCTestChannelPenalized: Bool,
119+
evidence: CoordinateTapTextInputEvidence?,
120+
point: CGPoint?,
121+
targetBundleId: String?
122+
) -> Bool {
123+
if xCTestChannelPenalized { return false }
124+
guard let evidence, let point, point.x.isFinite, point.y.isFinite,
125+
!evidence.targetBundleId.isEmpty, evidence.targetBundleId == targetBundleId,
126+
evidence.point.x == point.x, evidence.point.y == point.y
127+
else { return true }
128+
for input in evidence.textInputRects {
129+
let rect = CGRect(x: input.x, y: input.y, width: input.width, height: input.height)
130+
guard SnapshotGeometry.isPositiveFinite(rect) else { return true }
131+
if isCoordinateTextInputCandidate(enabled: true, frame: rect, point: point) { return true }
132+
}
133+
return false
118134
}
119135

120136
/// A synthesized `tap` step in a `sequence` is a standalone coordinate tap and follows its policy. On

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedGesturePolicyTests.swift‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -102,8 +102,8 @@ extension RunnerTests {
102102
}
103103

104104
func testCoordinateTapTextInputProbeSkipsPenalizedXCTestChannel() {
105-
XCTAssertTrue(shouldProbeCoordinateTapTextInput(xCTestChannelPenalized: false))
106-
XCTAssertFalse(shouldProbeCoordinateTapTextInput(xCTestChannelPenalized: true))
105+
XCTAssertTrue(shouldProbeCoordinateTapTextInput(xCTestChannelPenalized: false, evidence: nil, point: nil, targetBundleId: nil))
106+
XCTAssertFalse(shouldProbeCoordinateTapTextInput(xCTestChannelPenalized: true, evidence: nil, point: nil, targetBundleId: nil))
107107
}
108108

109109
func testOnlySynthesizedSequenceTapStepsTakeTheCoordinateTapPolicy() {
@@ -209,3 +209,34 @@ extension RunnerTests {
209209
}
210210
}
211211
#endif
212+
213+
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
214+
private struct CoordinateTapTextInputFixture: Decodable {
215+
let name: String
216+
let point: RunnerGesturePoint
217+
let dispatchPoint: RunnerGesturePoint?
218+
let activeBundleId: String
219+
let expectedEvidence: CoordinateTapTextInputEvidence?
220+
let probe: Bool
221+
}
222+
223+
extension RunnerTests {
224+
func testCoordinateTapTextInputAdmissionMatchesGoldenTable() throws {
225+
var root = URL(fileURLWithPath: #filePath)
226+
for _ in 0..<6 { root.deleteLastPathComponent() }
227+
let fixtures = try JSONDecoder().decode([CoordinateTapTextInputFixture].self, from:
228+
Data(contentsOf: root.appendingPathComponent("contracts/fixtures/coordinate-tap-text-input.json")))
229+
XCTAssertFalse(fixtures.isEmpty)
230+
for fixture in fixtures {
231+
let point = fixture.dispatchPoint ?? fixture.point
232+
XCTAssertEqual(shouldProbeCoordinateTapTextInput(
233+
xCTestChannelPenalized: false,
234+
evidence: fixture.expectedEvidence,
235+
point: CGPoint(x: point.x, y: point.y),
236+
targetBundleId: fixture.activeBundleId
237+
), fixture.probe, fixture.name)
238+
}
239+
XCTAssertTrue(shouldProbeCoordinateTapTextInput(xCTestChannelPenalized: false, evidence: nil, point: nil, targetBundleId: nil), "raw coordinates keep probing")
240+
}
241+
}
242+
#endif

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextInputProbeTests.swift‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,75 @@ extension RunnerTests {
8585
XCTAssertFalse(didRecordXCTestFailure(since: failures))
8686
}
8787

88+
@MainActor
89+
func testFreshCoordinateTapEvidenceSkipsProbeAndClearsTypingWitness() throws {
90+
app.launchArguments = ["--agent-device-text-entry-regression", "--agent-device-text-entry-tap-witness"]
91+
app.launch()
92+
defer {
93+
textInputProbeIssueForTesting = nil
94+
invalidateCachedTarget(reason: "unit_test_cleanup")
95+
app.terminate()
96+
}
97+
mainOwned.app = app
98+
mainOwned.bundleId = "com.callstack.agentdevice.runner"
99+
mainOwned.processIdentifier = try XCTUnwrap(Self.processIdentifier(of: app))
100+
clearSnapshotXCTestChannelPenalty(reason: "fresh-runner")
101+
let field = app.textFields["agent-device-hardware-keyboard-input"]
102+
XCTAssertTrue(field.waitForFixtureExistence(timeout: appExistenceTimeout))
103+
let fieldFrame = field.frame
104+
let fieldTap = try runnerCommandFixture(
105+
#"{"appBundleId":"com.callstack.agentdevice.runner","command":"tap","commandId":"tap-field-before-exclusion","x":\#(fieldFrame.midX),"y":\#(fieldFrame.midY),"synthesized":true}"#
106+
)
107+
XCTAssertTrue(try execute(command: fieldTap).ok)
108+
XCTAssertNotNil(textEntryTapWitness)
109+
let button = app.buttons["agent-device-tap-witness"]
110+
let frame = button.frame
111+
textInputProbeIssueForTesting = XCTIssue(type: .assertionFailure, compactDescription: "excluded point must not query text inputs")
112+
let tap = try runnerCommandFixture(
113+
#"{"appBundleId":"com.callstack.agentdevice.runner","command":"tap","commandId":"tap-excluded-point","x":\#(frame.midX),"y":\#(frame.midY),"synthesized":true,"coordinateTapTextInputEvidence":{"targetBundleId":"com.callstack.agentdevice.runner","point":{"x":\#(frame.midX),"y":\#(frame.midY)},"textInputRects":[{"x":\#(fieldFrame.minX),"y":\#(fieldFrame.minY),"width":\#(fieldFrame.width),"height":\#(fieldFrame.height)}]}}"#
114+
)
115+
let response = try execute(command: tap)
116+
XCTAssertTrue(response.ok, String(describing: response.error))
117+
XCTAssertNotNil(textInputProbeIssueForTesting, "skipping must bypass all optional text-input queries")
118+
textInputProbeIssueForTesting = nil
119+
XCTAssertNil(textEntryTapWitness, "a skipped probe must clear the preceding field's typing authorization")
120+
XCTAssertEqual(app.staticTexts["agent-device-tap-witness-presses"].label, "presses=1")
121+
try XCTSkipIf(isKeyboardVisible(app: app), "software keyboard is up; hidden-keyboard authorization cannot be exercised")
122+
let type = try runnerCommandFixture(#"{"appBundleId":"com.callstack.agentdevice.runner","command":"type","commandId":"type-after-exclusion","text":"must-not-type","textEntryMode":"append"}"#)
123+
let typed = try execute(command: type)
124+
XCTAssertFalse(typed.ok)
125+
XCTAssertEqual(typed.error?.code, "TEXT_INPUT_NOT_FOCUSED")
126+
XCTAssertEqual(field.value as? String, "")
127+
}
128+
129+
@MainActor
130+
func testCoordinateTapEvidenceContainingInputPreservesBareTypingWitness() throws {
131+
app.launchArguments = ["--agent-device-text-entry-regression"]
132+
app.launch()
133+
defer {
134+
invalidateCachedTarget(reason: "unit_test_cleanup")
135+
app.terminate()
136+
}
137+
let field = app.textFields["agent-device-hardware-keyboard-input"]
138+
XCTAssertTrue(field.waitForFixtureExistence(timeout: appExistenceTimeout))
139+
let frame = field.frame
140+
mainOwned.app = app
141+
mainOwned.bundleId = "com.callstack.agentdevice.runner"
142+
mainOwned.processIdentifier = try XCTUnwrap(Self.processIdentifier(of: app))
143+
clearSnapshotXCTestChannelPenalty(reason: "fresh-runner")
144+
let tap = try runnerCommandFixture(
145+
#"{"appBundleId":"com.callstack.agentdevice.runner","command":"tap","commandId":"tap-input-evidence","x":\#(frame.midX),"y":\#(frame.midY),"synthesized":true,"coordinateTapTextInputEvidence":{"targetBundleId":"com.callstack.agentdevice.runner","point":{"x":\#(frame.midX),"y":\#(frame.midY)},"textInputRects":[{"x":\#(frame.minX),"y":\#(frame.minY),"width":\#(frame.width),"height":\#(frame.height)}]}}"#
146+
)
147+
XCTAssertTrue(try execute(command: tap).ok)
148+
XCTAssertNotNil(textEntryTapWitness)
149+
try XCTSkipIf(isKeyboardVisible(app: app), "software keyboard is up; hidden-keyboard witness cannot be exercised")
150+
let type = try runnerCommandFixture(#"{"appBundleId":"com.callstack.agentdevice.runner","command":"type","commandId":"type-input-evidence","text":"evidence-witness","textEntryMode":"append"}"#)
151+
let response = try execute(command: type)
152+
XCTAssertTrue(response.ok, String(describing: response.error))
153+
XCTAssertEqual(response.data?.textEntryRoute, "xctest-element")
154+
XCTAssertEqual(field.value as? String, "evidence-witness")
155+
}
156+
88157
func testTextInputProbeContainmentExcludesRequiredReadsAndLaterIssues() {
89158
app.launchArguments = ["--agent-device-text-entry-regression"]
90159
app.launch()

0 commit comments

Comments
 (0)