Skip to content

Commit cc23f2b

Browse files
committed
fix(cloud): fail a witness-less cloud fill instead of reporting it filled
Review follow-ups on #1658. `not-observed` was still a success: it sent the keys and answered "Filled N chars", and nothing renders `textEntryReadiness` in default CLI output — so the exact silent success this branch exists to remove survived whenever focus never happened. A tap that raises no keyboard now fails with `text_entry_focus_not_observed` and sends no keys, leaving the field untouched rather than half-written, and the readiness vocabulary keeps only outcomes that describe a fill that did type. The readiness budget was advertised but not enforced at the request boundary: each keyboard probe inherited the client's 30s default, so one hung probe could hold a 2s wait for far longer. Probes now carry their own bound, threaded through the client as a per-request timeout override. The probe also swallowed every error as "this driver cannot answer", which degraded a dead session, an auth rejection, or a grid outage into a blind text entry. Only a positively classified unimplemented route counts as unsupported now — classified on the W3C error code rather than the status, since `unknown command` and `invalid session id` share HTTP 404 — and everything else propagates. The provider scenario proved request ordering against a stub that always accepted keys. Its fake now models the device: focus lands a beat after the tap, and keys arriving while the keyboard is down are accepted and dropped, exactly as an unfocused field does. The tests assert the field's own value, and both go red against the pre-fix `fill`. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015Zjbzf7HdziX9SzFpNP7WW
1 parent 2099757 commit cc23f2b

8 files changed

Lines changed: 373 additions & 73 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
## Unreleased
44

55
- Cloud iOS (BrowserStack, AWS Device Farm): `snapshot` and `diff` no longer fail with `SESSION_NOT_FOUND` on a live provider session (#1658). The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so the guard now applies to local Apple targets only. Relatedly, a cloud iOS `open com.example.app` now records that bundle id on the session — the provider path skips local app resolution (no simctl/devicectl reaches a hosted device), and used to drop an explicitly spelled bundle id along with it, leaving the session with no app identity at all. Opening a second bundle id replaces the first, matching the local path, where an explicitly spelled target always wins over the session's current app; deep links, display names, and bare `open` still keep the app already tracked.
6-
- Cloud `fill` (BrowserStack, AWS Device Farm) waits for the tapped field to take text-entry focus before sending its keys, instead of dispatching both in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The response now discloses `textEntryReadiness`: `keyboard-shown`, `settled-keyboard-up` (back-to-back fills into one form, where keyboard visibility cannot witness the new field taking focus, so the fill waits out the same readiness budget the Apple runner spends there), `settled-unknown` (the driver reports no keyboard state), or `not-observed` (the keys were sent, but no keyboard ever appeared).
6+
- Cloud `fill` (BrowserStack, AWS Device Farm) waits for the tapped field to take text-entry focus before sending its keys, instead of dispatching both in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The response now discloses `textEntryReadiness`: `keyboard-shown`, `settled-keyboard-up` (back-to-back fills into one form, where keyboard visibility cannot witness the new field taking focus, so the fill waits out the same readiness budget the Apple runner spends there), or `settled-unknown` (the driver implements no keyboard route at all). Breaking: when the tap raises no keyboard within that budget, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. A driver that cannot report keyboard state at all is unaffected (it blind-settles as before); only a positively classified unimplemented route counts as such, so a dead session, an auth rejection, or a grid outage now surfaces instead of degrading into a blind text entry.
77
- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
88
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
99
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.

‎packages/contracts/src/interactor-types.ts‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -110,10 +110,13 @@ export type TypeTextBackendResult = {
110110
* - `settled-keyboard-up`: the keyboard was ALREADY up (back-to-back fills into
111111
* one form), so its visibility says nothing about the NEW field. Entry waited
112112
* the full readiness budget, the same answer the Apple runner gives here.
113-
* - `settled-unknown`: the driver reports no keyboard state at all, so nothing
114-
* could be waited for. Entry followed a short blind settle.
115-
* - `not-observed`: the keyboard was down before the tap and never came up. The
116-
* keys were still sent, but nothing corroborates that a field received them.
113+
* - `settled-unknown`: the driver implements no keyboard route at all, so
114+
* nothing could be waited for. Entry followed a short blind settle.
115+
*
116+
* Every value describes a fill that DID send its keys. The fourth outcome — a
117+
* tap that raised no keyboard at all — is deliberately absent: reporting it
118+
* here would recreate #1658's silent success, so the interactor throws
119+
* (`text_entry_focus_not_observed`) without typing instead.
117120
*
118121
* Closed set: the cloud interactor is its only producer and the boundary that
119122
* narrows it (readFillBackendResult, src/core/dispatch-interactions.ts) drops a
@@ -123,7 +126,6 @@ export const CLOUD_TEXT_ENTRY_READINESS = [
123126
'keyboard-shown',
124127
'settled-keyboard-up',
125128
'settled-unknown',
126-
'not-observed',
127129
] as const;
128130

129131
export type CloudTextEntryReadiness = (typeof CLOUD_TEXT_ENTRY_READINESS)[number];
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
import assert from 'node:assert/strict';
2+
import { afterEach, test } from 'vitest';
3+
import { AppError } from '@agent-device/kernel/errors';
4+
import { WebDriverClient } from './webdriver-client.ts';
5+
6+
const realFetch = globalThis.fetch;
7+
8+
afterEach(() => {
9+
globalThis.fetch = realFetch;
10+
});
11+
12+
// #1658: `fill` blind-settles and types when the keyboard route is unsupported.
13+
// Anything else reaching that branch would degrade a real provider failure into
14+
// a blind text entry, so the split is pinned against the actual wire shapes.
15+
test('is_keyboard_shown reports an unimplemented route as unsupported', async () => {
16+
// Appium's own answer for a route the driver does not have.
17+
const client = await connectedClient(
18+
webDriverResponse(404, { value: { error: 'unknown command', message: 'Unknown command' } }),
19+
);
20+
21+
assert.equal(await client.isKeyboardShown(), 'unsupported');
22+
});
23+
24+
test('is_keyboard_shown reports a non-boolean answer as unsupported', async () => {
25+
const client = await connectedClient(webDriverResponse(200, { value: null }));
26+
27+
assert.equal(await client.isKeyboardShown(), 'unsupported');
28+
});
29+
30+
// The status is identical to an unimplemented route's, so a status-only test
31+
// would read a dead session as a missing feature and keep typing into it.
32+
test('is_keyboard_shown propagates a dead session, which shares the 404', async () => {
33+
const client = await connectedClient(
34+
webDriverResponse(404, {
35+
value: {
36+
error: 'invalid session id',
37+
message: 'A session is either terminated or not started',
38+
},
39+
}),
40+
);
41+
42+
await assert.rejects(client.isKeyboardShown(), (error: AppError) => {
43+
assert.match(error.message, /terminated or not started/);
44+
return true;
45+
});
46+
});
47+
48+
test('is_keyboard_shown propagates a grid failure', async () => {
49+
const client = await connectedClient(
50+
webDriverResponse(503, { value: { message: 'provider backend unavailable' } }),
51+
);
52+
53+
await assert.rejects(client.isKeyboardShown(), (error: AppError) => {
54+
assert.match(error.message, /provider backend unavailable/);
55+
return true;
56+
});
57+
});
58+
59+
// The readiness poll advertises a 2s budget; without the override each probe
60+
// would inherit the client's 30s default and one hung request could hold it
61+
// there. A stub that answers only the abort proves the caller's bound is the
62+
// one in force — were it ignored, this test would hang to its own timeout.
63+
test('is_keyboard_shown honors a caller timeout shorter than the client default', async () => {
64+
const client = await connectedClient(
65+
async (_input, init) =>
66+
await new Promise<Response>((_resolve, reject) => {
67+
init?.signal?.addEventListener('abort', () => reject(init.signal?.reason as Error));
68+
}),
69+
);
70+
71+
await assert.rejects(client.isKeyboardShown(50), (error: Error) => {
72+
assert.match(`${error.name} ${error.message}`, /timeout|abort/i);
73+
return true;
74+
});
75+
});
76+
77+
/** A client with a live session id, so `isKeyboardShown` reaches the session route. */
78+
async function connectedClient(respond: typeof globalThis.fetch): Promise<WebDriverClient> {
79+
globalThis.fetch = webDriverResponse(200, {
80+
value: { sessionId: 'wd-1', capabilities: {} },
81+
});
82+
const client = new WebDriverClient({
83+
clientVersion: '0.0.0-test',
84+
endpoint: 'http://cloud-webdriver.test/wd/hub/',
85+
requestPolicy: { timeoutMs: 30_000, retryAttempts: 0 },
86+
});
87+
await client.createSession({ platformName: 'iOS' });
88+
globalThis.fetch = respond;
89+
return client;
90+
}
91+
92+
function webDriverResponse(status: number, body: unknown): typeof globalThis.fetch {
93+
return async () =>
94+
new Response(JSON.stringify(body), {
95+
status,
96+
headers: { 'Content-Type': 'application/json' },
97+
});
98+
}

‎packages/provider-webdriver/src/webdriver-client.ts‎

Lines changed: 63 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,12 @@ type WebDriverResponse = {
6565

6666
type WebDriverRequestOverrides = {
6767
retryAttempts?: number;
68+
/**
69+
* Per-request transport bound, for callers whose own budget is far shorter
70+
* than the client's default. Without it a caller waiting 2s on a poll can be
71+
* held for the full default timeout by one hung request.
72+
*/
73+
timeoutMs?: number;
6874
};
6975

7076
export class WebDriverClient {
@@ -143,22 +149,29 @@ export class WebDriverClient {
143149
}
144150

145151
/**
146-
* Whether the software keyboard is up. Polled between a text field tap and
147-
* the keys it should receive, so it must stay a single cheap round trip: no
148-
* retries, and a driver that does not implement the route reports through the
149-
* thrown error rather than a guessed boolean (callers decide what an
150-
* unanswerable keyboard state means for them).
152+
* Whether the software keyboard is up, or `unsupported` when this driver has
153+
* no such route. Polled between a text field tap and the keys it should
154+
* receive, so it stays one cheap round trip — no retries, and `timeoutMs`
155+
* lets the caller bound it by its own budget rather than the client default.
156+
*
157+
* The unsupported/failed split matters: a caller that treats every error as
158+
* "this driver cannot answer" would silently degrade a dead session, an auth
159+
* rejection, or a grid outage into a blind text entry. Only a route the
160+
* driver does not implement — or one that answers with a non-boolean — is
161+
* reported as unsupported; everything else throws.
151162
*/
152-
async isKeyboardShown(): Promise<boolean> {
153-
const value = await this.sessionRequest('GET', '/appium/device/is_keyboard_shown', undefined, {
154-
retryAttempts: 0,
155-
});
156-
if (typeof value !== 'boolean') {
157-
throw new AppError('COMMAND_FAILED', 'WebDriver keyboard-shown response was not a boolean', {
158-
valueType: typeof value,
163+
async isKeyboardShown(timeoutMs?: number): Promise<boolean | 'unsupported'> {
164+
let value: unknown;
165+
try {
166+
value = await this.sessionRequest('GET', '/appium/device/is_keyboard_shown', undefined, {
167+
retryAttempts: 0,
168+
...(timeoutMs === undefined ? {} : { timeoutMs }),
159169
});
170+
} catch (error) {
171+
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
172+
throw error;
160173
}
161-
return value;
174+
return typeof value === 'boolean' ? value : 'unsupported';
162175
}
163176

164177
async back(): Promise<void> {
@@ -228,9 +241,10 @@ export class WebDriverClient {
228241
): Promise<unknown> {
229242
let lastError: unknown;
230243
const retryAttempts = overrides?.retryAttempts ?? this.requestPolicy.retryAttempts;
244+
const timeoutMs = overrides?.timeoutMs ?? this.requestPolicy.timeoutMs;
231245
for (let attempt = 0; attempt <= retryAttempts; attempt += 1) {
232246
try {
233-
return await this.requestValueOnce(method, path, body);
247+
return await this.requestValueOnce(method, path, body, timeoutMs);
234248
} catch (error) {
235249
lastError = error;
236250
if (!isRetriableWebDriverError(error) || attempt >= retryAttempts) {
@@ -242,7 +256,12 @@ export class WebDriverClient {
242256
throw lastError;
243257
}
244258

245-
private async requestValueOnce(method: string, path: string, body?: unknown): Promise<unknown> {
259+
private async requestValueOnce(
260+
method: string,
261+
path: string,
262+
body: unknown,
263+
timeoutMs: number,
264+
): Promise<unknown> {
246265
const response = await fetch(new URL(trimLeadingSlash(path), this.endpoint), {
247266
method,
248267
headers: {
@@ -251,7 +270,7 @@ export class WebDriverClient {
251270
...this.headers,
252271
},
253272
body: body === undefined ? undefined : JSON.stringify(body),
254-
signal: AbortSignal.timeout(this.requestPolicy.timeoutMs),
273+
signal: AbortSignal.timeout(timeoutMs),
255274
});
256275
const text = await response.text();
257276
const payload = text ? parseJsonResponse(text) : {};
@@ -348,6 +367,34 @@ function webdriverError(status: number, payload: unknown): AppError {
348367
return new AppError('COMMAND_FAILED', message, { status, response: payload });
349368
}
350369

370+
/**
371+
* A route this driver does not implement, as opposed to one that failed.
372+
*
373+
* Classified from the W3C error code, NOT from the HTTP status: `unknown
374+
* command` and `invalid session id` are both 404, so a status test would read a
375+
* dead session as a missing feature — exactly the confusion that turns a broken
376+
* session into a blind text entry. Only 405/501 are unambiguous enough to stand
377+
* on their own. A 5xx, an auth rejection, or a timeout is a real failure.
378+
*/
379+
const UNIMPLEMENTED_WEBDRIVER_STATUSES = new Set([405, 501]);
380+
const UNIMPLEMENTED_WEBDRIVER_ERRORS = new Set(['unknown command', 'unknown method']);
381+
382+
function isUnimplementedWebDriverRoute(error: unknown): boolean {
383+
if (!(error instanceof AppError)) return false;
384+
const status = error.details?.status;
385+
if (typeof status === 'number' && UNIMPLEMENTED_WEBDRIVER_STATUSES.has(status)) return true;
386+
return UNIMPLEMENTED_WEBDRIVER_ERRORS.has(readWebDriverErrorCode(error).toLowerCase());
387+
}
388+
389+
function readWebDriverErrorCode(error: AppError): string {
390+
const response = error.details?.response;
391+
const value =
392+
response && typeof response === 'object' ? (response as { value?: unknown }).value : undefined;
393+
const code =
394+
value && typeof value === 'object' ? (value as { error?: unknown }).error : undefined;
395+
return typeof code === 'string' ? code : '';
396+
}
397+
351398
function isRetriableWebDriverError(error: unknown): boolean {
352399
if (error instanceof AppError) {
353400
const status = error.details?.status;

0 commit comments

Comments
 (0)