Skip to content

Commit ef89178

Browse files
claudethymikee
authored andcommitted
fix(cloud): unblock iOS snapshot and gate cloud fill on text-entry focus (#1658)
Two bugs isolated to the cloud-webdriver iOS path. `snapshot`/`diff` refused every capture on a live BrowserStack session with SESSION_NOT_FOUND, instantly and without a driver round trip. The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so it now applies to local Apple targets only. The session was empty in the first place because the provider open path skips local app resolution wholesale — no simctl/devicectl reaches a hosted device — and dropped an explicitly spelled bundle id along with it. A dotted, non-deep-link target is the bundle id under the same convention resolveIosApp applies locally, so a cloud `open com.example.app` now records it. `fill` tapped and sent its keys in back-to-back requests. A WebView input — an OAuth page in a Safari view controller — takes first responder asynchronously, so the keys landed with nothing focused while the command still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The cloud interactor now waits on the same signal the Apple runner uses, the software keyboard going from hidden to shown after its tap, and discloses what it observed as `textEntryReadiness` so a fill with no witness cannot pass for a filled field. Where keyboard visibility cannot witness the focus move — back-to-back fills into one form, the shape that failed most often — it spends the runner's full readiness budget rather than racing the app with a short settle. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015Zjbzf7HdziX9SzFpNP7WW
1 parent a158434 commit ef89178

13 files changed

Lines changed: 643 additions & 7 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
## Unreleased
44

55
- `agent-device help workflow` is now a compact ~8KB card instead of a ~41KB dump; the same depth still exists, split into `help scripting` (save-script, secret-safe fills, batch JSON, replay divergence/repair, recording) and `help gestures` (multi-touch shapes and platform quirks), plus a few paragraphs folded into the topics that already owned the subject (`help debugging`, `help physical-device`, `help validate`). Every `help <topic>` first line is now `agent-device <version> — <topic>` so an agent can read the installed version from its mandatory first help read instead of a separate `agent-device --version` call.
6-
6+
- Cloud iOS (BrowserStack, AWS Device Farm): `snapshot` and `diff` no longer fail with `SESSION_NOT_FOUND` on a live provider session (#1658). The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so the guard now applies to local Apple targets only. Relatedly, a cloud iOS `open com.example.app` now records that bundle id on the session — the provider path skips local app resolution (no simctl/devicectl reaches a hosted device), and used to drop an explicitly spelled bundle id along with it, leaving the session with no app identity at all.
7+
- Cloud `fill` (BrowserStack, AWS Device Farm) waits for the tapped field to take text-entry focus before sending its keys, instead of dispatching both in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The response now discloses `textEntryReadiness`: `keyboard-shown`, `settled-keyboard-up` (back-to-back fills into one form, where keyboard visibility cannot witness the new field taking focus, so the fill waits out the same readiness budget the Apple runner spends there), `settled-unknown` (the driver reports no keyboard state), or `not-observed` (the keys were sent, but no keyboard ever appeared).
78
- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
89
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
910
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.

‎packages/contracts/src/facades/interaction.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,10 +106,16 @@ export type {
106106
SettleParams,
107107
SettleTailEntry,
108108
} from '../interaction.ts';
109-
export { MAESTRO_NON_HITTABLE_FALLBACK_MESSAGE, TEXT_ENTRY_ROUTES } from '../interactor-types.ts';
109+
export {
110+
CLOUD_TEXT_ENTRY_READINESS,
111+
MAESTRO_NON_HITTABLE_FALLBACK_MESSAGE,
112+
TEXT_ENTRY_ROUTES,
113+
} from '../interactor-types.ts';
110114
export type {
115+
CloudTextEntryReadiness,
111116
ElementSelectorKey,
112117
ElementSelectorTapOptions,
118+
FillBackendResult,
113119
Interactor,
114120
RunnerCallOptions,
115121
RunnerContext,

‎packages/contracts/src/interactor-types.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,45 @@ export type TypeTextBackendResult = {
9898
textEntryRoute?: TextEntryRoute;
9999
};
100100

101+
/**
102+
* What a cloud-WebDriver `fill` could establish about its target taking text
103+
* entry focus before it sent the keys (#1658). The local Apple runner owns this
104+
* discipline in Swift and reports it as a `textEntryRoute`; the cloud path has
105+
* no runner, only the driver's keyboard-visibility signal, so it names what it
106+
* observed instead of which channel it typed through:
107+
*
108+
* - `keyboard-shown`: the software keyboard went from hidden to shown after our
109+
* tap — positive evidence the tap moved focus to a text input.
110+
* - `settled-keyboard-up`: the keyboard was ALREADY up (back-to-back fills into
111+
* one form), so its visibility says nothing about the NEW field. Entry waited
112+
* the full readiness budget, the same answer the Apple runner gives here.
113+
* - `settled-unknown`: the driver reports no keyboard state at all, so nothing
114+
* could be waited for. Entry followed a short blind settle.
115+
* - `not-observed`: the keyboard was down before the tap and never came up. The
116+
* keys were still sent, but nothing corroborates that a field received them.
117+
*
118+
* Closed set: the cloud interactor is its only producer and the boundary that
119+
* narrows it (readFillBackendResult, src/core/dispatch-interactions.ts) drops a
120+
* value it cannot name.
121+
*/
122+
export const CLOUD_TEXT_ENTRY_READINESS = [
123+
'keyboard-shown',
124+
'settled-keyboard-up',
125+
'settled-unknown',
126+
'not-observed',
127+
] as const;
128+
129+
export type CloudTextEntryReadiness = (typeof CLOUD_TEXT_ENTRY_READINESS)[number];
130+
131+
/**
132+
* What `Interactor.fill` reports back about the entry it performed. Only the
133+
* cloud-WebDriver interactor populates `textEntryReadiness`; the Apple runner
134+
* carries its own equivalent in Swift and every other platform fills blind.
135+
*/
136+
export type FillBackendResult = {
137+
textEntryReadiness?: CloudTextEntryReadiness;
138+
};
139+
101140
export type SnapshotOptions = BaseSnapshotOptions & {
102141
appBundleId?: string;
103142
signal?: AbortSignal;

‎packages/provider-webdriver/src/webdriver-client.ts‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,25 @@ export class WebDriverClient {
142142
});
143143
}
144144

145+
/**
146+
* Whether the software keyboard is up. Polled between a text field tap and
147+
* the keys it should receive, so it must stay a single cheap round trip: no
148+
* retries, and a driver that does not implement the route reports through the
149+
* thrown error rather than a guessed boolean (callers decide what an
150+
* unanswerable keyboard state means for them).
151+
*/
152+
async isKeyboardShown(): Promise<boolean> {
153+
const value = await this.sessionRequest('GET', '/appium/device/is_keyboard_shown', undefined, {
154+
retryAttempts: 0,
155+
});
156+
if (typeof value !== 'boolean') {
157+
throw new AppError('COMMAND_FAILED', 'WebDriver keyboard-shown response was not a boolean', {
158+
valueType: typeof value,
159+
});
160+
}
161+
return value;
162+
}
163+
145164
async back(): Promise<void> {
146165
await this.sessionRequest('POST', '/back');
147166
}

‎packages/provider-webdriver/src/webdriver-interactor.test.ts‎

Lines changed: 112 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,121 @@
11
import assert from 'node:assert/strict';
2-
import { test } from 'vitest';
2+
import { test, vi } from 'vitest';
33
import { buildGesturePlan } from '@agent-device/contracts/interaction';
4+
import { AppError } from '@agent-device/kernel/errors';
45
import { createCloudWebDriverCapabilities } from './capabilities.ts';
56
import type { WebDriverClient, W3CActionSequence } from './webdriver-client.ts';
67
import { createWebDriverInteractor } from './webdriver-interactor.ts';
78

9+
// #1658: `fill` used to send its keys in the request right after the tap. A
10+
// WebView input takes first responder asynchronously, so on a web login form
11+
// the keys landed with nothing focused and `fill` still reported success.
12+
test('fill withholds keys until the tapped field raises the keyboard', async () => {
13+
const world = createTextEntryWorld();
14+
// Hidden at the tap, then up two polls later: the transition our tap caused.
15+
world.keyboardShown = [false, false, false, true];
16+
17+
const result = await runFill(world);
18+
19+
assert.equal(result?.textEntryReadiness, 'keyboard-shown');
20+
assert.deepEqual(world.transcript, [
21+
'keyboard',
22+
'tap',
23+
'keyboard',
24+
'keyboard',
25+
'keyboard',
26+
'keys',
27+
]);
28+
});
29+
30+
test('fill settles when the keyboard was already up before the tap', async () => {
31+
const world = createTextEntryWorld();
32+
// Back-to-back fills (email then password): a keyboard that never went down
33+
// cannot witness focus moving to the NEW field, so no poll can prove anything
34+
// — polling it would only re-observe evidence about the previous field.
35+
world.keyboardShown = [true];
36+
37+
const result = await runFill(world);
38+
39+
assert.equal(result?.textEntryReadiness, 'settled-keyboard-up');
40+
assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']);
41+
});
42+
43+
test('fill settles when the driver cannot report keyboard state', async () => {
44+
const world = createTextEntryWorld();
45+
world.keyboardSupported = false;
46+
47+
const result = await runFill(world);
48+
49+
// Evidence, never a precondition: an unanswerable keyboard must not fail the fill.
50+
assert.equal(result?.textEntryReadiness, 'settled-unknown');
51+
assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']);
52+
});
53+
54+
test('fill still types when no keyboard ever appears, and says so', async () => {
55+
const world = createTextEntryWorld();
56+
world.keyboardShown = [false];
57+
58+
const result = await runFill(world);
59+
60+
assert.equal(result?.textEntryReadiness, 'not-observed');
61+
assert.equal(world.transcript.at(-1), 'keys');
62+
});
63+
64+
/**
65+
* Drives `fill` on a fake clock: the readiness wait is real production time on
66+
* a device and must never be real time here (docs/agents/testing.md). Advancing
67+
* past the whole readiness budget is safe for every case — once readiness
68+
* resolves there are no pending timers left to fire.
69+
*/
70+
async function runFill(world: ReturnType<typeof createTextEntryWorld>) {
71+
vi.useFakeTimers();
72+
try {
73+
const interactor = createWebDriverInteractor({
74+
client: world.client,
75+
backend: 'xctest',
76+
capabilities: createCloudWebDriverCapabilities({ provider: 'test', platform: 'ios' }),
77+
});
78+
const pending = interactor.fill(12, 24, '[email protected]');
79+
await vi.advanceTimersByTimeAsync(5_000);
80+
return await pending;
81+
} finally {
82+
vi.useRealTimers();
83+
}
84+
}
85+
86+
/**
87+
* Records the request order `fill` produces. The ordering IS the fix: a `keys`
88+
* entry that follows `tap` with no keyboard evidence between them is the bug.
89+
*/
90+
function createTextEntryWorld() {
91+
const world = {
92+
transcript: [] as string[],
93+
keyboardShown: [] as boolean[],
94+
keyboardSupported: true,
95+
client: undefined as unknown as WebDriverClient,
96+
};
97+
world.client = {
98+
performActions: async () => {
99+
world.transcript.push('tap');
100+
},
101+
releaseActions: async () => {},
102+
sendKeys: async () => {
103+
world.transcript.push('keys');
104+
},
105+
isKeyboardShown: async () => {
106+
world.transcript.push('keyboard');
107+
if (!world.keyboardSupported) {
108+
throw new AppError('COMMAND_FAILED', 'Unknown command: is_keyboard_shown');
109+
}
110+
// The last programmed reading is what the keyboard stays at.
111+
return world.keyboardShown.length > 1
112+
? world.keyboardShown.shift()!
113+
: world.keyboardShown[0]!;
114+
},
115+
} as unknown as WebDriverClient;
116+
return world;
117+
}
118+
8119
test('endpoint plans become one timed W3C pointer move', async () => {
9120
const performed: W3CActionSequence[][] = [];
10121
let released = false;

‎packages/provider-webdriver/src/webdriver-interactor.ts‎

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import type { DeviceRotation } from '@agent-device/contracts/device';
22
import type {
33
BackMode,
4+
CloudTextEntryReadiness,
45
GesturePlan,
56
Interactor,
67
ScreenshotOptions,
@@ -24,6 +25,21 @@ import { scrollFrameFromWebDriverSource } from './webdriver-scroll-frame.ts';
2425
import { parseWebDriverSource } from './webdriver-source.ts';
2526
import { setWebDriverOrientation } from './webdriver-orientation.ts';
2627

28+
/**
29+
* How long a tapped field gets to take text entry focus before `fill` types
30+
* anyway. Mirrors the Apple runner's `TextEntryTiming.readinessTimeout`
31+
* (RunnerTests+TextEntry.swift), which owns this discipline locally.
32+
*/
33+
const TEXT_ENTRY_READINESS_TIMEOUT_MS = 2_000;
34+
const TEXT_ENTRY_READINESS_POLL_MS = 100;
35+
/** The blind wait when the driver reports no keyboard state at all — see `awaitTextEntryReadiness`. */
36+
const TEXT_ENTRY_BLIND_SETTLE_MS = 350;
37+
38+
/** Global-timer based, so text-entry readiness can be exercised on a fake clock. */
39+
function sleep(ms: number): Promise<void> {
40+
return new Promise((resolve) => setTimeout(resolve, ms));
41+
}
42+
2743
export type WebDriverInteractorOptions = {
2844
client: WebDriverClient;
2945
backend: Extract<SnapshotResult['backend'], 'android' | 'xctest'>;
@@ -147,9 +163,14 @@ class WebDriverInteractor implements Interactor {
147163
_delayMs?: number,
148164
): Promise<Record<string, unknown>> {
149165
this.requireSupport('fill');
166+
// #1658: read the keyboard BEFORE the tap. Only a hidden -> shown
167+
// transition proves that OUR tap moved focus; a keyboard that was already
168+
// up says nothing about which field owns first responder now.
169+
const keyboardBeforeTap = await this.tryReadKeyboardShown();
150170
await this.tap(x, y);
171+
const textEntryReadiness = await this.awaitTextEntryReadiness(keyboardBeforeTap);
151172
await this.type(text);
152-
return { backend: 'webdriver', x, y, text };
173+
return { backend: 'webdriver', x, y, text, textEntryReadiness };
153174
}
154175

155176
async scroll(
@@ -256,6 +277,54 @@ class WebDriverInteractor implements Interactor {
256277
this.unsupported('settings');
257278
}
258279

280+
/**
281+
* The cloud twin of the local Apple runner's focus -> readiness -> type
282+
* pipeline (RunnerTests+TextEntry.swift). A WebView input does not take first
283+
* responder synchronously with the tap: the page has to process the touch and
284+
* raise the keyboard. `fill` used to send its keys in the very next request,
285+
* so on a slow web login form they landed with nothing focused — reported as
286+
* "Filled N chars" while the field kept its placeholder. That is also why
287+
* tapping and filling as two separate commands was the reliable workaround:
288+
* only the round trip between them gave the field time to focus.
289+
*/
290+
private async awaitTextEntryReadiness(
291+
keyboardBeforeTap: boolean | undefined,
292+
): Promise<CloudTextEntryReadiness> {
293+
// The driver answers nothing about the keyboard, so no wait can learn
294+
// anything. Keep a blind path cheap: settle briefly, then type.
295+
if (keyboardBeforeTap === undefined) {
296+
await sleep(TEXT_ENTRY_BLIND_SETTLE_MS);
297+
return 'settled-unknown';
298+
}
299+
// The keyboard was ALREADY up — back-to-back fills into one form, the shape
300+
// that failed most often in #1658. Its visibility cannot witness focus
301+
// moving to the NEW field, so polling it would resolve instantly on
302+
// evidence about the PREVIOUS one. The Apple runner spends its whole
303+
// readiness budget on exactly this case rather than race the app with a
304+
// short settle; the cloud path has even more reason to, because its keys go
305+
// to whatever holds first responder rather than to a scoped element.
306+
if (keyboardBeforeTap) {
307+
await sleep(TEXT_ENTRY_READINESS_TIMEOUT_MS);
308+
return 'settled-keyboard-up';
309+
}
310+
const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS;
311+
while (Date.now() < deadline) {
312+
await sleep(TEXT_ENTRY_READINESS_POLL_MS);
313+
if ((await this.tryReadKeyboardShown()) === true) return 'keyboard-shown';
314+
}
315+
// The tap opened no keyboard. Still send the keys — a text view without a
316+
// software keyboard (hardware keyboard attached, or a non-editable target
317+
// the caller means to type at) is a real case, and refusing would turn a
318+
// recoverable miss into a hard failure — but say so, so the caller is not
319+
// told "filled" by a fill that had no witness.
320+
return 'not-observed';
321+
}
322+
323+
/** Keyboard state is evidence, never a precondition: a driver that cannot answer must not fail the fill. */
324+
private async tryReadKeyboardShown(): Promise<boolean | undefined> {
325+
return await this.client.isKeyboardShown().catch(() => undefined);
326+
}
327+
259328
private async pointerGesture(name: string, actions: W3CPointerAction[]): Promise<void> {
260329
await this.client.performActions([touchPointer(name, actions)]);
261330
// Some Appium grids accept W3C actions but reject DELETE /actions. A failed

‎src/core/dispatch-interactions.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
import {
22
assertExclusiveScrollDistanceInputs,
3+
CLOUD_TEXT_ENTRY_READINESS,
34
getClickButtonValidationError,
45
honoredScrollDurationMs,
56
MAESTRO_NON_HITTABLE_FALLBACK_MESSAGE,
67
normalizeScrollDurationMs,
78
parseScrollDirection,
89
resolveClickButton,
910
type ClickButton,
11+
type CloudTextEntryReadiness,
12+
type FillBackendResult,
1013
type Interactor,
1114
type RunnerCallOptions,
1215
type ScrollCommandOptions,
@@ -125,10 +128,28 @@ export async function handleFillCommand(
125128
...(result?.maestroNonHittableCoordinateFallbackUsed === true
126129
? { maestroNonHittableCoordinateFallbackUsed: true }
127130
: {}),
131+
...readFillBackendResult(result),
128132
...successText(formatTextLengthMessage('Filled', text)),
129133
};
130134
}
131135

136+
/**
137+
* The interactor result is untrusted; this is the one place a `fill` response
138+
* becomes the typed {@link FillBackendResult}. Only the cloud-WebDriver
139+
* interactor reports a readiness today, and a value this boundary cannot name
140+
* is dropped rather than forwarded.
141+
*/
142+
function readFillBackendResult(result: Record<string, unknown> | void): FillBackendResult {
143+
const readiness = result ? result.textEntryReadiness : undefined;
144+
return isCloudTextEntryReadiness(readiness) ? { textEntryReadiness: readiness } : {};
145+
}
146+
147+
function isCloudTextEntryReadiness(value: unknown): value is CloudTextEntryReadiness {
148+
return (
149+
typeof value === 'string' && (CLOUD_TEXT_ENTRY_READINESS as readonly string[]).includes(value)
150+
);
151+
}
152+
132153
async function handleDirectElementSelectorFill(
133154
interactor: Interactor,
134155
selector: NonNullable<DispatchContext['directElementSelector']>,

0 commit comments

Comments
 (0)