Skip to content

Commit aa8f5a7

Browse files
committed
fix: stop stamping recovered iOS captures truncated; confirm Android alert dismissal
Two CI flake families on main and PRs since 2026-09-03. iOS Smoke, `is absent ... capture was truncated` (7 of 13 failures): the runner's stampedSnapshotPayload set `truncated: true` on every non-healthy capture, so a complete private-AX tree taken while the XCTest channel was penalized as slow (the normal state on a loaded CI host) was reported as truncated. Nothing consumed that until the strict absence assertion (#2245) refused truncated captures. `truncated` now tracks completeness only: payload truncation, a depth-limited capture, or a sparse terminal payload. The E2E conformance helper asserted the old conflation and now asserts `truncated === false`; a runner unit test pins the new contract and joins the targeted list in ios.yml. Android Smoke, `get text id="automation-alert-result"` selector miss (5 of 5 failures): #2260 replaced a polling wait with a one-shot read right after `alert dismiss`, and Android's `alert accept|dismiss` returned as soon as the button was pressed, while the dialog window was still the only thing in the accessibility tree. They now poll until the same dialog is gone (a different alert taking its place counts as dismissed), bounded by the existing action budget, else fail with "did not dismiss the visible alert" like the iOS runner already does.
1 parent 7a2d48d commit aa8f5a7

7 files changed

Lines changed: 173 additions & 17 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,7 @@ jobs:
214214
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXInteractiveFiltersLoginLikeHiddenDrawer \
215215
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDecodedPreferredBackendReachesOptionsAndApplicablePlan \
216216
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSparsePayloadReasonMatrix \
217+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testStampedPayloadTruncationTracksCompletenessNotRecoveryProvenance \
217218
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledNeedsEnoughSamples \
218219
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledTrueWhenWindowMatches \
219220
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledFalseOnMidWindowMismatch \

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,16 @@
22

33
## Unreleased
44

5+
- Fixed: iOS snapshots no longer report `truncated: true` merely because a later backend produced
6+
them. The runner stamped every recovered capture as truncated — including a complete private-AX
7+
tree taken while the XCTest channel was penalized as slow — so a strict `is absent` / `wait absent`
8+
refused it with "capture was truncated" on loaded CI hosts. `truncated` now tracks completeness
9+
only: payload truncation, a depth-limited capture, or a sparse terminal payload.
10+
- Fixed: Android `alert accept` / `alert dismiss` return only once the dialog has left the
11+
accessibility tree (a different alert taking its place counts as dismissed), matching the iOS
12+
runner's re-check. Previously they returned right after the button press, so the next read could
13+
still see only the dialog window. A dialog that stays visible past the action budget now fails with
14+
`alert <action> did not dismiss the visible alert`.
515
- Added strict `wait absent <selector> [timeoutMs]` polling for zero selector matches. Incomplete,
616
sparse, truncated, scoped, depth-limited, and Android unreadable captures cannot prove absence;
717
deadline diagnostics retain typed capture evidence and stable first-match details (#2236).

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift‎

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -645,7 +645,11 @@ extension RunnerTests {
645645
// Legacy human text for older daemons that read message instead of snapshotQuality.
646646
message: Self.legacyQualityMessage(quality) ?? payload.message,
647647
nodes: payload.nodes,
648-
truncated: payload.truncated == true || state != "healthy" || capture.effectiveDepth != nil,
648+
// Completeness, never provenance: a whole tree that a later backend produced (state
649+
// "recovered") stays untruncated, so strict absence reads can trust it. Only a real cap
650+
// (payload truncation, a depth-limited private AX capture) or a sparse terminal payload
651+
// is truncated.
652+
truncated: payload.truncated == true || state == "sparse" || capture.effectiveDepth != nil,
649653
qualityPayload: capture.qualityPayload.flatMap { quality in
650654
guard let nodes = quality.nodes else { return nil }
651655
return SnapshotQualityPayload(nodes: nodes, truncated: quality.truncated == true)
@@ -886,6 +890,46 @@ extension RunnerTests {
886890
XCTAssertEqual(payload.nodes?.count, 1)
887891
}
888892

893+
func testStampedPayloadTruncationTracksCompletenessNotRecoveryProvenance() {
894+
let complete = SnapshotBackendCapture(
895+
payload: DataPayload(
896+
nodes: [
897+
planTestNode(index: 0, type: "Application", label: "App"),
898+
planTestNode(index: 1, type: "Button", label: "Open", parentIndex: 0),
899+
],
900+
truncated: false
901+
),
902+
effectiveDepth: nil
903+
)
904+
let deferred: (reason: String, code: String) = (
905+
"XCTest-backed snapshot tiers were deferred after recent slow accessibility work", "deferred"
906+
)
907+
908+
// The CI signature behind `is absent ... capture was truncated`: a complete private AX
909+
// tree selected while the XCTest channel is penalized is whole, and must say so.
910+
let recovered = stampedSnapshotPayload(
911+
complete, backend: .privateAX, state: "recovered", reason: deferred)
912+
XCTAssertEqual(recovered.snapshotQuality?.state, "recovered")
913+
XCTAssertEqual(recovered.truncated, false)
914+
915+
let depthLimited = stampedSnapshotPayload(
916+
SnapshotBackendCapture(payload: complete.payload, effectiveDepth: 56),
917+
backend: .privateAX, state: "recovered", reason: deferred)
918+
XCTAssertEqual(depthLimited.truncated, true)
919+
920+
let cappedPayload = stampedSnapshotPayload(
921+
SnapshotBackendCapture(
922+
payload: DataPayload(nodes: complete.payload.nodes ?? [], truncated: true),
923+
effectiveDepth: nil),
924+
backend: .recursiveTree, state: "healthy", reason: nil)
925+
XCTAssertEqual(cappedPayload.truncated, true)
926+
927+
let sparse = stampedSnapshotPayload(
928+
complete, backend: .querySweep, state: "sparse",
929+
reason: ("snapshot returned no semantic controls or content", "sparse-tree"))
930+
XCTAssertEqual(sparse.truncated, true)
931+
}
932+
889933
func testSnapshotQualityCarriesUnscopedQualityPayload() {
890934
let quality = DataPayload(
891935
nodes: [planTestNode(index: 0, type: "Application", label: "App")],

‎packages/platform-android/src/__tests__/alert.test.ts‎

Lines changed: 78 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,22 @@ const runAndroidAdb = vi.fn(async (_device: DeviceInfo, _args: string[]) => ({
99
stderr: '',
1010
}));
1111
vi.mock('../adb.ts', () => ({ runAndroidAdb }));
12+
// The dismissal re-check polls at the contract interval; the clock is the assertion, not the wait.
13+
vi.mock('@agent-device/host-kit/retry', () => ({ sleep: async () => {} }));
1214

1315
const { handleAndroidAlert } = await import('../alert.ts');
1416

17+
const dialog = [
18+
node(0, 'android.app.AlertDialog'),
19+
text(1, 'Automation confirmation', 'android:id/alertTitle'),
20+
button(2, 'Cancel', 'android:id/button2', { x: 210, y: 612 }),
21+
];
22+
23+
/** The dialog is in the tree until the button press lands, then gone — the real timeline. */
24+
function dialogUntilPressed(nodes = dialog) {
25+
return async () => (runAndroidAdb.mock.calls.length === 0 ? nodes : []);
26+
}
27+
1528
const device: DeviceInfo = {
1629
platform: 'android',
1730
id: 'emulator-5554',
@@ -23,11 +36,7 @@ const device: DeviceInfo = {
2336
test('dismissing a button alert records the tapped button and its coordinates', async () => {
2437
runAndroidAdb.mockClear();
2538
const result = await handleAndroidAlert(device, 'dismiss', {
26-
captureNodes: async () => [
27-
node(0, 'android.app.AlertDialog'),
28-
text(1, 'Automation confirmation', 'android:id/alertTitle'),
29-
button(2, 'Cancel', 'android:id/button2', { x: 210, y: 612 }),
30-
],
39+
captureNodes: dialogUntilPressed(),
3140
});
3241

3342
assert.deepEqual(result, {
@@ -52,11 +61,11 @@ test('dismissing a button alert records the tapped button and its coordinates',
5261
test('accepting a button alert records the tapped button and its coordinates', async () => {
5362
runAndroidAdb.mockClear();
5463
const result = await handleAndroidAlert(device, 'accept', {
55-
captureNodes: async () => [
64+
captureNodes: dialogUntilPressed([
5665
node(0, 'android.app.AlertDialog'),
5766
text(1, 'Automation confirmation', 'android:id/alertTitle'),
5867
button(2, 'OK', 'android:id/button1', { x: 52, y: 612 }),
59-
],
68+
]),
6069
});
6170

6271
assert.equal(result.kind, 'alertHandled');
@@ -66,14 +75,74 @@ test('accepting a button alert records the tapped button and its coordinates', a
6675
test('a fallback Back dismissal (no matching button) carries no coordinates', async () => {
6776
runAndroidAdb.mockClear();
6877
const result = await handleAndroidAlert(device, 'dismiss', {
69-
captureNodes: async () => [
78+
captureNodes: dialogUntilPressed([
7079
node(0, 'android.app.AlertDialog'),
7180
text(1, 'Automation confirmation', 'android:id/alertTitle'),
72-
],
81+
]),
7382
});
7483

7584
assert.equal(result.kind, 'alertHandled');
7685
assert.ok(result.kind === 'alertHandled' && !('coordinates' in result));
7786
assert.equal(result.kind === 'alertHandled' ? result.button : undefined, 'Back');
7887
assert.deepEqual(runAndroidAdb.mock.calls[0]?.[1], ['shell', 'input', 'keyevent', '4']);
7988
});
89+
90+
test('dismiss returns only after the dialog has left the tree', async () => {
91+
runAndroidAdb.mockClear();
92+
// Pre-press lookup, then two captures that still show the closing dialog, then the app.
93+
const captures = [dialog, dialog, dialog, []];
94+
let reads = 0;
95+
const result = await handleAndroidAlert(device, 'dismiss', {
96+
captureNodes: async () => captures[Math.min(reads++, captures.length - 1)] ?? [],
97+
});
98+
99+
assert.equal(result.kind, 'alertHandled');
100+
assert.equal(reads, 4);
101+
assert.equal(runAndroidAdb.mock.calls.length, 1);
102+
});
103+
104+
test('a different alert replacing the pressed one counts as dismissed', async () => {
105+
runAndroidAdb.mockClear();
106+
const followUp = [
107+
node(0, 'android.app.AlertDialog'),
108+
text(1, 'Discard changes?', 'android:id/alertTitle'),
109+
button(2, 'Keep', 'android:id/button2', { x: 210, y: 612 }),
110+
];
111+
const result = await handleAndroidAlert(device, 'dismiss', {
112+
captureNodes: async () => (runAndroidAdb.mock.calls.length === 0 ? dialog : followUp),
113+
});
114+
115+
assert.equal(result.kind, 'alertHandled');
116+
assert.equal(
117+
result.kind === 'alertHandled' ? result.alert.title : undefined,
118+
'Automation confirmation',
119+
);
120+
});
121+
122+
test('dismiss fails when the dialog is still visible after the action budget', async () => {
123+
runAndroidAdb.mockClear();
124+
vi.useFakeTimers({ now: 0, toFake: ['Date'] });
125+
try {
126+
let reads = 0;
127+
await assert.rejects(
128+
handleAndroidAlert(device, 'dismiss', {
129+
captureNodes: async () => {
130+
// Every post-press capture costs wall clock; the dialog never leaves.
131+
if (reads++ > 0) vi.setSystemTime(Date.now() + 700);
132+
return dialog;
133+
},
134+
}),
135+
(error: unknown) =>
136+
error instanceof Error &&
137+
error.message === 'alert dismiss did not dismiss the visible alert' &&
138+
(error as { code?: string }).code === 'COMMAND_FAILED',
139+
);
140+
assert.equal(runAndroidAdb.mock.calls.length, 1);
141+
assert.ok(
142+
reads >= 4,
143+
`expected the re-check to poll until the budget expired, got ${reads} reads`,
144+
);
145+
} finally {
146+
vi.useRealTimers();
147+
}
148+
});

‎packages/platform-android/src/alert.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ async function handleAndroidAlertAction(
103103
const button = chooseAndroidAlertButton(candidate.buttons, action);
104104
if (button) {
105105
await pressAndroid(device, button.x, button.y);
106+
await confirmAndroidAlertDismissed(candidate.alert, action, captureNodes);
106107
return buildAndroidAlertHandledResponse(action, candidate.alert, button.label, {
107108
x: button.x,
108109
y: button.y,
@@ -111,6 +112,7 @@ async function handleAndroidAlertAction(
111112

112113
if (action === 'dismiss') {
113114
await backAndroid(device);
115+
await confirmAndroidAlertDismissed(candidate.alert, action, captureNodes);
114116
return buildAndroidAlertHandledResponse(action, candidate.alert, 'Back');
115117
}
116118

@@ -120,6 +122,35 @@ async function handleAndroidAlertAction(
120122
});
121123
}
122124

125+
/**
126+
* `alert accept|dismiss` means the dialog is gone, not that a button was pressed: the next
127+
* command reads the app, and a capture that lands while the dialog window is still up sees
128+
* only the dialog. A different alert taking its place counts as dismissed. Bounded by the
129+
* same budget the pre-press lookup uses; iOS's runner applies the same re-check.
130+
*/
131+
async function confirmAndroidAlertDismissed(
132+
dismissed: AndroidAlertInfo,
133+
action: 'accept' | 'dismiss',
134+
captureNodes: AndroidAlertOptions['captureNodes'],
135+
): Promise<void> {
136+
const start = Date.now();
137+
for (;;) {
138+
const current = await readAndroidAlertCandidate(captureNodes);
139+
if (!current || !sameAndroidAlert(current.alert, dismissed)) return;
140+
if (Date.now() - start >= ALERT_ACTION_RETRY_MS) {
141+
throw new AppError('COMMAND_FAILED', `alert ${action} did not dismiss the visible alert`, {
142+
alert: dismissed,
143+
hint: 'The alert button was pressed but the dialog is still visible. Inspect alert get --json, then press the visible button by label/ref or retry.',
144+
});
145+
}
146+
await sleep(ALERT_POLL_INTERVAL_MS);
147+
}
148+
}
149+
150+
function sameAndroidAlert(left: AndroidAlertInfo, right: AndroidAlertInfo): boolean {
151+
return left.title === right.title && left.buttons.join('\u0000') === right.buttons.join('\u0000');
152+
}
153+
123154
async function pollAndroidAlertCandidate(
124155
captureNodes: AndroidAlertOptions['captureNodes'],
125156
timeoutMs: number,

‎test/integration/ios-simulator-e2e/snapshot-backend-conformance.ts‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -82,13 +82,14 @@ export function assertSnapshotBackendConformance(
8282
quality?.state === 'healthy' || quality?.state === 'recovered',
8383
`${backend} capture must have a non-sparse quality verdict: ${JSON.stringify(quality)}`,
8484
);
85-
// The existing wire contract marks any recovered capture as truncated, including a complete
86-
// private-AX payload selected after the XCTest channel was deferred. Assert that relationship
87-
// instead of conflating recovery provenance with missing fixture controls.
85+
// `truncated` is a completeness fact, never recovery provenance: a complete capture stays
86+
// untruncated even when a later backend produced it after the XCTest channel was deferred
87+
// (state "recovered"). The fixture screen fits every backend's budget, so truncation here is
88+
// a wire regression, not missing fixture controls.
8889
assert.equal(
8990
snapshot.truncated,
90-
quality.state !== 'healthy',
91-
`${backend} quality/truncation flags disagree: ${JSON.stringify(quality)}`,
91+
false,
92+
`${backend} reported a truncated capture of the fixture screen: ${JSON.stringify(quality)}`,
9293
);
9394
assert.ok(
9495
snapshot.nodes.length >= fixture.minimumNodeCount,

‎test/integration/smoke-ios-snapshot-backend-conformance.test.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,9 @@ test('snapshot backend conformance rejects every promised control invariant', ()
4545
/must have a non-sparse quality verdict/,
4646
);
4747
expectFailure(
48-
'recovered/truncated mismatch',
48+
'truncated fixture capture',
4949
{ ...base, truncated: true },
50-
/quality\/truncation/,
50+
/reported a truncated capture/,
5151
);
5252
expectFailure('minimum node count', { ...base, nodes: base.nodes.slice(0, 2) }, /too few nodes/);
5353
expectFailure(

0 commit comments

Comments
 (0)