Skip to content

Commit 28c6cf1

Browse files
committed
fix(runner): tell an empty fill point from a blind one before the tap
The pre-tap lookup returned nil for both, so one refusal carried two claims. coordinateTapTextInputIdentityAt now answers input, absent, or unavailable; absent refuses with NO_TEXT_INPUT_AT_POINT and unavailable refuses with TEXT_INPUT_PROBE_UNAVAILABLE, both before the focus tap. The hint and docs name every route that reaches the guard, including ref, selector and replay fills, whose aim point is the centre of the rect they resolved. Fixes #1721
1 parent 3a164d1 commit 28c6cf1

12 files changed

Lines changed: 132 additions & 29 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
## Unreleased
44

55
- Add `settings mobile-data on|off` for adb-backed Android targets. It switches mobile data through `svc data` and reports the `mobileData` state stored in the global `mobile_data` row; multi-SIM read-back is not covered yet. Every other target refuses it before anything runs, and `--app` is refused because the switch is device-wide.
6+
- On iOS, a `fill` that runs under a degraded XCTest snapshot channel now aims at a point, including ref, selector and replay fills. When the runner identifies no text input under that point, it refuses with `NO_TEXT_INPUT_AT_POINT` (`error.details.reason: no_text_input_at_point`, `dispatched: no`) before dispatching its focus tap, and no field changes; when the input lookup itself cannot answer, it refuses with `TEXT_INPUT_PROBE_UNAVAILABLE` (`error.details.reason: text_input_probe_unavailable`). Previously such a fill cleared and typed through the app-level first responder, so it could rewrite a field the caller never named, or report a clear failure after emptying it. A ref or selector fill whose rect centre misses the real field (an outer container, a non-input node) now needs a point inside the input's own rect.
67
- Limrun iOS snapshots no longer fail when the instance reports a numeric or boolean accessibility value (the system photo picker's tabs report `1`/`0`): the provider seam normalizes variant scalars to text, the way the Apple tree does. Limrun nodes now also publish `selected: true` for nodes carrying the `Selected` trait, so snapshot output and `selected:` selectors can tell a picked photo from an unpicked one.
78
- `settings` refuses a setting the target does not serve before anything changes, with `UNSUPPORTED_OPERATION`, `settings <setting> is not supported on this device`, and the target's `error.details.reason` (such as `unsupported-platform-leaf`). This replaces the macOS `INVALID_ARGS` refusal and the Apple `setting-unsupported-on-leaf` reason, and now also covers settings that used to reach the device first, such as Android `reset-keychain` or Apple `animations`.
89
- `settings` refuses, before touching the device, a state its setting does not accept (`Invalid <setting> state: …`), a permission action or mode its target does not take (such as `permission deny accessibility`), and a stray extra argument (such as `clear-app-state <app> <extra>`). The CLI and the daemon accept the same state spellings, in any casing or surrounding whitespace.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextInputProbe.swift‎

Lines changed: 34 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,44 @@ extension RunnerTests {
3838
}
3939
}
4040

41+
/// What the point's text-input lookup can say before a coordinate gesture is dispatched: the
42+
/// input it found, no input at the point, or nothing. A lookup that cannot answer is neither of the
43+
/// first two, and must not be read as one of them.
44+
enum CoordinateTextInputResolution {
45+
case input(TextInputAtPoint)
46+
case absent
47+
case unavailable
48+
}
49+
4150
/// The input `coordinateTapTextInputAt` finds, with the identity it carried then, read inside the
42-
/// same issue containment. Nil when the probe finds none or cannot answer.
43-
func coordinateTapTextInputIdentityAt(app: XCUIApplication, x: Double, y: Double) -> TextInputAtPoint? {
44-
let probed = containingTextInputProbeIssues(fallback: nil) { shouldStop -> TextInputAtPoint? in
45-
guard let element = queryTextInputs(app: app, point: CGPoint(x: x, y: y), shouldStop: shouldStop).first,
46-
case .input(let identity) = probeTextEntryInput(element)
51+
/// same issue containment, split into what the lookup can name so a refusal can tell an empty
52+
/// point from a blind one.
53+
func coordinateTapTextInputIdentityAt(
54+
app: XCUIApplication,
55+
x: Double,
56+
y: Double
57+
) -> CoordinateTextInputResolution {
58+
let probed = containingTextInputProbeIssues(fallback: CoordinateTextInputResolution.unavailable) {
59+
shouldStop -> CoordinateTextInputResolution in
60+
guard let element = queryTextInputs(app: app, point: CGPoint(x: x, y: y), shouldStop: shouldStop).first
4761
else {
48-
return nil
62+
return .absent
4963
}
50-
return TextInputAtPoint(element: element, identity: identity)
64+
switch probeTextEntryInput(element) {
65+
case .input(let identity):
66+
return .input(TextInputAtPoint(element: element, identity: identity))
67+
case .noMatch:
68+
return .absent
69+
case .unavailable:
70+
return .unavailable
71+
}
72+
}
73+
switch probed {
74+
case .success(let resolution):
75+
return resolution
76+
case .failure:
77+
return .unavailable
5178
}
52-
guard case .success(let input) = probed else { return nil }
53-
return input
5479
}
5580

5681
/// Whether `input`'s handle still resolves to the input it named when found, read inside the

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TypeExecution.swift‎

Lines changed: 29 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,17 @@ extension RunnerTests {
2828
xCTestChannelPenalized: xCTestChannelPenalized
2929
), let x = command.x, let y = command.y {
3030
// Resolved before the tap: the focus it causes can move the field away from this point.
31-
let inputAtPoint = coordinateTapTextInputIdentityAt(app: activeApp, x: x, y: y)
32-
// No input at the point means no target to bind the replacement to. Dispatching it anyway
33-
// clears and types through the app-level first responder, so a fill aimed at empty space
34-
// rewrites whichever field an earlier focus left active.
35-
guard let inputAtPoint else {
31+
// A replacement with no input at the point has nothing to bind to: its clear and its typed
32+
// text would reach the app-level first responder, so an empty point and a lookup that cannot
33+
// answer both refuse here, each under its own reason.
34+
let inputAtPoint: TextInputAtPoint
35+
switch coordinateTapTextInputIdentityAt(app: activeApp, x: x, y: y) {
36+
case .input(let input):
37+
inputAtPoint = input
38+
case .absent:
3639
return noTextInputAtPointResponse()
40+
case .unavailable:
41+
return textInputProbeUnavailableResponse()
3742
}
3843
let policyKind = SynthesizedGesturePolicyKind.coordinateTap
3944
let context = synthesizedCoordinateContext(
@@ -156,17 +161,31 @@ extension RunnerTests {
156161
return Response(ok: true, data: data)
157162
}
158163

159-
/// The refusal of a coordinate `fill` whose point names no text input, answered before the focus
160-
/// tap is dispatched so nothing runs. Without an input at the point the synthesized replacement has
161-
/// no element to bind to, so its clear and its typed text would reach the app-level first
162-
/// responder and overwrite whatever field currently holds focus.
164+
/// The refusal of a coordinate `fill` whose point identified no text input, answered before the
165+
/// focus tap is dispatched so nothing runs. Without an input at the point the synthesized
166+
/// replacement has no element to bind to, so its clear and its typed text would reach the app-level
167+
/// first responder and overwrite whatever field currently holds focus.
163168
private func noTextInputAtPointResponse() -> Response {
164169
Response(
165170
ok: false,
166171
error: ErrorPayload(
167172
code: RunnerWireErrorCode.noTextInputAtPoint,
168173
message: "no text input identified at the provided coordinates",
169-
hint: "A coordinate fill replaces the text of the input under the point, so the point must sit inside one. Run snapshot -i, then fill the field by ref or selector, or aim at a point inside its rect. The runner also reports this when its input lookup cannot answer, so re-run snapshot before concluding the point is empty."
174+
hint: "A fill replaces the text of the input under its point, so the point must sit inside one — for a ref or selector fill, that point is the centre of the rect it resolved to, which an outer container or a non-input node does not give you. Run snapshot -i, pick the input node itself, and aim at a point inside its rect."
175+
)
176+
)
177+
}
178+
179+
/// The refusal of a coordinate `fill` whose point the input lookup could not read, answered before
180+
/// the focus tap so nothing runs. The channel is degraded, so the runner cannot tell whether an
181+
/// input is there; typing would reach the app-level first responder, which is what #1721 refuses.
182+
private func textInputProbeUnavailableResponse() -> Response {
183+
Response(
184+
ok: false,
185+
error: ErrorPayload(
186+
code: RunnerWireErrorCode.textInputProbeUnavailable,
187+
message: "the runner could not read the text input at the provided coordinates",
188+
hint: "The accessibility lookup for the point could not answer on the degraded channel, so the runner dispatched nothing and no field changed — this does not mean the point has no input. Retry the fill, or run snapshot -i and fill the input node by ref once its capture reads back."
170189
)
171190
)
172191
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests.swift‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,14 @@ final class RunnerTests: XCTestCase {
3333
/// gate; `MAIN_THREAD_TIMEOUT` is emitted by the transport when a command trips the execution
3434
/// watchdog, so the daemon can tell "the main thread is now occupied" from a generic failure.
3535
/// `APP_NOT_RUNNING` refuses a read whose session app is not running.
36-
/// `NO_TEXT_INPUT_AT_POINT` refuses a coordinate fill whose point names no text input.
36+
/// `NO_TEXT_INPUT_AT_POINT` refuses a coordinate fill whose point identified no text input.
37+
/// `TEXT_INPUT_PROBE_UNAVAILABLE` refuses one whose input lookup could not answer on the degraded
38+
/// channel, so the runner cannot tell whether an input is there.
3739
enum RunnerWireErrorCode {
3840
static let mainThreadTimeout = "MAIN_THREAD_TIMEOUT"
3941
static let appNotRunning = "APP_NOT_RUNNING"
4042
static let noTextInputAtPoint = "NO_TEXT_INPUT_AT_POINT"
43+
static let textInputProbeUnavailable = "TEXT_INPUT_PROBE_UNAVAILABLE"
4144
}
4245

4346
static let springboardBundleId = "com.apple.springboard"

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+CoordinateTextEntryTests.swift‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,26 @@ import XCTest
22

33
extension RunnerTests {
44
#if AGENT_DEVICE_RUNNER_UNIT_TESTS && os(iOS)
5+
/// The input the coordinate probe finds at the point, failing the test when it finds none or
6+
/// cannot answer. A fixture that does not expose its own field is a broken lane, not a refusal.
7+
@MainActor
8+
func requireTextInputAtPoint(app: XCUIApplication, x: Double, y: Double) throws -> TextInputAtPoint {
9+
switch coordinateTapTextInputIdentityAt(app: app, x: x, y: y) {
10+
case .input(let input):
11+
return input
12+
case .absent:
13+
return try XCTUnwrap(
14+
nil as TextInputAtPoint?,
15+
"the probe found no input at the fixture's own field"
16+
)
17+
case .unavailable:
18+
return try XCTUnwrap(
19+
nil as TextInputAtPoint?,
20+
"the probe could not read the fixture's own field"
21+
)
22+
}
23+
}
24+
525
@MainActor
626
func testPenalizedCoordinateTapOnNonTextControlDoesNotAuthorizeBareType() throws {
727
app.launchArguments = ["--agent-device-text-entry-regression"]

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedTextEntryTests.swift‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -222,7 +222,7 @@ extension RunnerTests {
222222
let field = try focusSynthesizedReplacementField()
223223
defer { tearDownSynthesizedReplacementField() }
224224
let frame = field.frame
225-
let found = try XCTUnwrap(coordinateTapTextInputIdentityAt(app: app, x: frame.midX, y: frame.midY))
225+
let found = try requireTextInputAtPoint(app: app, x: frame.midX, y: frame.midY)
226226
let cases: [(dropped: Int, failure: TextEntryFailure?)] = [
227227
(2, nil),
228228
(SynthesizedTextPlan.Step.maxClearPassCount, .clearNotObserved),
@@ -422,8 +422,8 @@ extension RunnerTests {
422422
/// The pre-tap lookup runs on a channel already penalized for failing XCTest reads. A lookup that
423423
/// fails used to record the failure, which fails the command and ends the runner, and left the
424424
/// fill to the point. Containment stays: no recorded failure. What changed is what the fill then
425-
/// does with a point it cannot name an input for: it refuses rather than typing through the
426-
/// first responder (#1721).
425+
/// does with a point it cannot read: it refuses under the probe-unavailable reason, which is not a
426+
/// claim that the point has no input (#1721).
427427
@MainActor
428428
func testSynthesizedReplacementFailsClosedOnAFailedLookupOfTheInputUnderThePoint() throws {
429429
let textField = try focusSynthesizedReplacementField()
@@ -440,7 +440,7 @@ extension RunnerTests {
440440

441441
XCTAssertNil(textInputProbeIssueForTesting, "the fill made no lookup")
442442
XCTAssertFalse(response.ok)
443-
XCTAssertEqual(response.error?.code, "NO_TEXT_INPUT_AT_POINT")
443+
XCTAssertEqual(response.error?.code, "TEXT_INPUT_PROBE_UNAVAILABLE")
444444
XCTAssertEqual(String(describing: textField.value ?? ""), "")
445445
}
446446
#endif

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextEntryPolicyTests.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -702,7 +702,7 @@ extension RunnerTests {
702702
element: nil,
703703
refreshPoint: CGPoint(x: frame.midX, y: frame.midY),
704704
prefersFocusedElement: false,
705-
inputAtRefreshPoint: try XCTUnwrap(coordinateTapTextInputIdentityAt(app: app, x: frame.midX, y: frame.midY))
705+
inputAtRefreshPoint: try requireTextInputAtPoint(app: app, x: frame.midX, y: frame.midY)
706706
),
707707
text: "abc",
708708
delaySeconds: 0.001,

‎contracts/fixtures/dispatch-disclosure.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,12 @@
199199
"trigger": "structured runner reply with code NO_TEXT_INPUT_AT_POINT: a coordinate fill refused before its focus tap because the runner identified no text input under the point",
200200
"dispatched": "no"
201201
},
202+
{
203+
"id": "ios-runner.reply.TEXT_INPUT_PROBE_UNAVAILABLE",
204+
"producer": "ios-runner",
205+
"trigger": "structured runner reply with code TEXT_INPUT_PROBE_UNAVAILABLE: a coordinate fill refused before its focus tap because the pre-tap input lookup could not answer",
206+
"dispatched": "no"
207+
},
202208
{
203209
"id": "ios-runner.reply.APP_SCREEN_WINDOW_UNRESOLVED",
204210
"producer": "ios-runner",

‎packages/platform-apple/src/runner/__tests__/index-dispatch-disclosure.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,8 @@ const DRIVERS: Record<string, () => Promise<unknown>> = {
169169
replyFailure('SCROLL_KEYBOARD_OCCLUDES_SURFACE'),
170170
'ios-runner.reply.ALERT_NOT_FOUND': () => replyFailure('ALERT_NOT_FOUND'),
171171
'ios-runner.reply.NO_TEXT_INPUT_AT_POINT': () => replyFailure('NO_TEXT_INPUT_AT_POINT'),
172+
'ios-runner.reply.TEXT_INPUT_PROBE_UNAVAILABLE': () =>
173+
replyFailure('TEXT_INPUT_PROBE_UNAVAILABLE'),
172174
'ios-runner.reply.APP_SCREEN_WINDOW_UNRESOLVED': () =>
173175
replyFailure('APP_SCREEN_WINDOW_UNRESOLVED'),
174176
'ios-runner.reply.APP_SCREEN_UNRESOLVED': () => replyFailure('APP_SCREEN_UNRESOLVED'),

‎packages/platform-apple/src/runner/__tests__/runner-error-classification.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import { AppError, createRequestCanceledError } from '@agent-device/kernel/error
44
import {
55
NO_TEXT_INPUT_AT_POINT_RUNNER_CODE,
66
RUNNER_SCREEN_CAPTURE_REFUSAL_RUNNER_CODES,
7+
TEXT_INPUT_PROBE_UNAVAILABLE_RUNNER_CODE,
78
classifyRunnerReportedError,
89
} from '../runner-contract.ts';
910
import {
@@ -117,6 +118,21 @@ test('a coordinate fill refused for a point with no input keeps its code and pub
117118
assert.equal(isRetryableRunnerError(untyped), false);
118119
});
119120

121+
// The two coordinate-fill refusals answer different questions and must never collapse into one
122+
// another: an empty point is a fact about the screen, a blind lookup is a fact about the channel.
123+
test('a coordinate fill refused on an unreadable point reports its own code, not no-input', () => {
124+
assert.equal(TEXT_INPUT_PROBE_UNAVAILABLE_RUNNER_CODE, 'TEXT_INPUT_PROBE_UNAVAILABLE');
125+
const refusal = classifyRunnerReportedError(TEXT_INPUT_PROBE_UNAVAILABLE_RUNNER_CODE);
126+
assert.equal(refusal.code, 'TEXT_INPUT_PROBE_UNAVAILABLE');
127+
assert.equal(refusal.details.reason, 'text_input_probe_unavailable');
128+
assert.equal(refusal.details.dispatched, 'no');
129+
assert.notEqual(
130+
refusal.details.reason,
131+
classifyRunnerReportedError(NO_TEXT_INPUT_AT_POINT_RUNNER_CODE).details.reason,
132+
);
133+
assert.equal(isRetryableRunnerError(new AppError(refusal.code, '', refusal.details)), false);
134+
});
135+
120136
test('retryable requires an AppError with COMMAND_FAILED', () => {
121137
assert.equal(isRetryableRunnerError(new Error('fetch failed')), false);
122138
assert.equal(isRetryableRunnerError(new AppError('DEVICE_NOT_FOUND', 'fetch failed')), false);

0 commit comments

Comments
 (0)