Skip to content

Commit aa8c42e

Browse files
thymikeeclaude
andauthored
fix(snapshot): name the captured Android app in the snapshot header (callstack#4499)
* fix(snapshot): name the captured Android app in the snapshot header After a failed open the launcher was in front, but the Page/App header named the session app, so an agent read launcher nodes as Settings. Android nodes carry their window package; when no app node belongs to the session app the header (appName/appBundleId in the response) names the captured package. The session keeps its app. Fixes callstack#4483 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(snapshot): keep the header identity test in its own file The snapshot test file passed its 1000-line lint limit. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(snapshot): name the header app from Android application windows Address review: scanning node packages let system UI, framework dialogs or an IME win the header when the session app was absent. The Android helper classifier already reads window types, so it now lists the captured non-system application windows (androidSnapshot.applicationWindowPackages), and the header names the topmost one only when none belongs to the session app. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(snapshot): order Android app windows by layer and name one without a session app Address review: the helper lists windows in getWindows() order, which is not z-order, so the header could name an app behind another. Windows now sort by window layer. A session with no app names the topmost captured app instead of printing no header, and the docs describe the list as deduplicated package names. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(snapshot): name the header app from the foreground application window Address review: a list of every application window could keep a stale session-app window that Android leaves behind after a transition. The helper classifier now reports one foreground window package, active or focused at the highest layer, using the presentation's foreground-window predicate, and the header names it. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(snapshot): key the unchanged acknowledgement on the header app Address review: the unchanged check compared the session app while the header named the foreground app, so a repeated snapshot of the launcher could read as unchanged against Settings. The check now uses the app the header names. A test fixture no longer writes a raw ampersand into an XML attribute. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(snapshot): keep main's identity for the unchanged acknowledgement Address review: comparing the header app with the session app disabled the unchanged acknowledgement for as long as another app stayed in front. The acknowledgement already requires the same tree as the last stored capture, which is the tree the header describes, so the identity check is back to main's. A test pins the acknowledgement under the foreign app's header. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 1229eb7 commit aa8c42e

10 files changed

Lines changed: 262 additions & 15 deletions

File tree

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import { createLocalArtifactAdapter } from '@agent-device/host-kit/local-artifact-adapter';
4+
import { createAgentDevice } from '../../../runtime.ts';
5+
import { localCommandPolicy } from '@agent-device/contracts/command-policy';
6+
import type { CommandSessionStore } from '@agent-device/contracts/runtime-contract';
7+
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
8+
9+
async function snapshotWithForeground(
10+
foregroundWindowPackage: string | undefined,
11+
sessionApp: { appName: string; appBundleId: string } | undefined = {
12+
appName: 'Settings',
13+
appBundleId: 'com.android.settings',
14+
},
15+
) {
16+
let stored: Parameters<CommandSessionStore['set']>[0] | undefined;
17+
const device = createAgentDevice({
18+
backend: {
19+
platform: 'android',
20+
captureSnapshot: async () => ({
21+
snapshot: makeSnapshotState([{ index: 0, depth: 0, type: 'TextView', label: 'Gmail' }], {
22+
backend: 'android',
23+
}),
24+
androidSnapshot: {
25+
backend: 'android-helper',
26+
...(foregroundWindowPackage ? { foregroundWindowPackage } : {}),
27+
},
28+
...sessionApp,
29+
}),
30+
},
31+
artifacts: createLocalArtifactAdapter(),
32+
sessions: {
33+
get: () => undefined,
34+
set: (record) => {
35+
stored = record;
36+
},
37+
},
38+
policy: localCommandPolicy(),
39+
});
40+
const result = await device.capture.snapshot({ session: 'default', interactiveOnly: true });
41+
return { result, stored };
42+
}
43+
44+
test('an Android snapshot names the captured foreground app when it is not the session app', async () => {
45+
const { result, stored } = await snapshotWithForeground('com.google.android.apps.nexuslauncher');
46+
47+
// A foreign app has no label of its own here, so the header shows its package.
48+
assert.equal(result.appName, 'com.google.android.apps.nexuslauncher');
49+
assert.equal(result.appBundleId, 'com.google.android.apps.nexuslauncher');
50+
assert.equal(stored?.appBundleId, 'com.android.settings');
51+
});
52+
53+
test('an Android snapshot keeps the session app label when the session app is in front', async () => {
54+
const { result } = await snapshotWithForeground('com.android.settings');
55+
56+
assert.equal(result.appName, 'Settings');
57+
assert.equal(result.appBundleId, 'com.android.settings');
58+
});
59+
60+
test('a snapshot without a foreground window fact names the session app', async () => {
61+
const { result } = await snapshotWithForeground(undefined);
62+
63+
assert.equal(result.appName, 'Settings');
64+
assert.equal(result.appBundleId, 'com.android.settings');
65+
});
66+
67+
test('an Android snapshot in a session with no app names the captured foreground app', async () => {
68+
const { result } = await snapshotWithForeground(
69+
'com.google.android.apps.nexuslauncher',
70+
undefined,
71+
);
72+
73+
assert.equal(result.appName, 'com.google.android.apps.nexuslauncher');
74+
assert.equal(result.appBundleId, 'com.google.android.apps.nexuslauncher');
75+
});
76+
77+
test('a repeated unchanged snapshot of a foreign foreground app is acknowledged under its header', async () => {
78+
const snapshot = makeSnapshotState([{ index: 0, depth: 0, type: 'TextView', label: 'Gmail' }], {
79+
backend: 'android',
80+
});
81+
let stored: Parameters<CommandSessionStore['set']>[0] | undefined = {
82+
name: 'default',
83+
appBundleId: 'com.android.settings',
84+
};
85+
const device = createAgentDevice({
86+
backend: {
87+
platform: 'android',
88+
captureSnapshot: async () => ({
89+
snapshot,
90+
androidSnapshot: {
91+
backend: 'android-helper',
92+
foregroundWindowPackage: 'com.google.android.apps.nexuslauncher',
93+
},
94+
appBundleId: 'com.android.settings',
95+
}),
96+
},
97+
artifacts: createLocalArtifactAdapter(),
98+
sessions: {
99+
get: () => stored,
100+
set: (record) => {
101+
stored = record;
102+
},
103+
},
104+
policy: localCommandPolicy(),
105+
});
106+
107+
await device.capture.snapshot({ session: 'default' });
108+
const result = await device.capture.snapshot({ session: 'default' });
109+
110+
// The acknowledgement compares against the last stored tree, the launcher's, which the header names.
111+
assert.equal(result.appBundleId, 'com.google.android.apps.nexuslauncher');
112+
assert.ok(result.unchanged);
113+
});

‎packages/command-runtime/src/commands/capture/runtime/snapshot.ts‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -234,15 +234,22 @@ function nextSnapshotSession(
234234
};
235235
}
236236

237+
/**
238+
* The app the response header names: the foreground application window the Android snapshot
239+
* helper captured, such as the launcher after a failed open, else the session app. The session
240+
* keeps its app.
241+
*/
237242
function snapshotAppFields(capture: SnapshotCapture): {
238243
appName?: string;
239244
appBundleId?: string;
240245
} {
241-
const appName = capture.result.appName ?? capture.session?.appName;
242-
const appBundleId = capture.result.appBundleId ?? capture.session?.appBundleId;
246+
const sessionApp = capture.result.appBundleId ?? capture.session?.appBundleId;
247+
const shownApp = capture.annotations.androidSnapshot?.foregroundWindowPackage ?? sessionApp;
248+
const appName =
249+
shownApp === sessionApp ? (capture.result.appName ?? capture.session?.appName) : shownApp;
243250
return {
244-
...(appName || appBundleId ? { appName: appName ?? appBundleId } : {}),
245-
...(appBundleId ? { appBundleId } : {}),
251+
...(appName || shownApp ? { appName: appName ?? shownApp } : {}),
252+
...(shownApp ? { appBundleId: shownApp } : {}),
246253
};
247254
}
248255

‎packages/contracts/src/snapshot-types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,11 @@ export type AndroidSnapshotBackendMetadata = {
7979
cacheInvalidated?: boolean;
8080
captureMode?: string;
8181
systemSurfaceOnly?: boolean;
82+
/**
83+
* Package of the foreground application window the snapshot helper captured: active or focused,
84+
* at the highest layer, and not owned by the system. Absent when the helper saw none.
85+
*/
86+
foregroundWindowPackage?: string;
8287
windowCount?: number;
8388
/**
8489
* `AccessibilityWindowInfo` types of the listed windows the helper could not serialize because

‎packages/platform-android/src/__tests__/snapshot-content-recovery.test.ts‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,88 @@ test('rejects system chrome inherited under an empty application window', () =>
270270
expect(decision?.diagnostics.helperNonSystemMeaningfulNodeCount).toBe(0);
271271
});
272272

273+
function foregroundWindowPackageOf(windows: string[]): string | undefined {
274+
const result = classifyAndroidHelperContent(helperXml(windows), {
275+
backend: 'android-helper',
276+
nodeCount: windows.length * 2,
277+
rootPresent: true,
278+
windowCount: windows.length,
279+
captureMode: 'interactive-windows',
280+
});
281+
return result.outcome === 'unusable' ? undefined : result.foregroundWindowPackage;
282+
}
283+
284+
function window(params: {
285+
windowType: number;
286+
packageName: string;
287+
text: string;
288+
windowLayer: number;
289+
windowActive?: boolean;
290+
}): string {
291+
const { text, ...windowAttributes } = params;
292+
return node({
293+
...windowAttributes,
294+
className: 'android.widget.FrameLayout',
295+
children: [
296+
node({ text, packageName: params.packageName, className: 'android.widget.TextView' }),
297+
],
298+
});
299+
}
300+
301+
test('names the active application window, not a stale one listed first or a system window above it', () => {
302+
expect(
303+
foregroundWindowPackageOf([
304+
window({
305+
windowType: 3,
306+
packageName: 'com.android.systemui',
307+
text: '12:00',
308+
windowLayer: 9,
309+
windowActive: true,
310+
}),
311+
window({
312+
windowType: 2,
313+
packageName: 'com.google.android.inputmethod.latin',
314+
text: 'Space',
315+
windowLayer: 8,
316+
}),
317+
// A stale session-app window Android kept after the transition: inactive, lower layer.
318+
window({
319+
windowType: 1,
320+
packageName: 'com.android.settings',
321+
text: 'Network and internet',
322+
windowLayer: 1,
323+
}),
324+
window({
325+
windowType: 1,
326+
packageName: 'com.google.android.apps.nexuslauncher',
327+
text: 'Gmail',
328+
windowLayer: 2,
329+
windowActive: true,
330+
}),
331+
]),
332+
).toBe('com.google.android.apps.nexuslauncher');
333+
});
334+
335+
test('names no foreground app when only a framework window is active', () => {
336+
expect(
337+
foregroundWindowPackageOf([
338+
window({
339+
windowType: 1,
340+
packageName: 'android',
341+
text: 'Allow?',
342+
windowLayer: 3,
343+
windowActive: true,
344+
}),
345+
window({
346+
windowType: 1,
347+
packageName: 'com.android.settings',
348+
text: 'Network and internet',
349+
windowLayer: 1,
350+
}),
351+
]),
352+
).toBeUndefined();
353+
});
354+
273355
test('returns a meaningful active system surface (notification shade) faithfully', () => {
274356
const result = classifyAndroidHelperContent(
275357
helperXml([
@@ -526,12 +608,14 @@ function node(options: {
526608
packageName: string;
527609
className: string;
528610
windowType?: number;
611+
windowLayer?: number;
529612
windowActive?: boolean;
530613
/** Nested children, so chrome leaves can sit inside their status/nav-bar container as on a device. */
531614
children?: string[];
532615
}): string {
533616
const windowAttrs =
534617
(options.windowType === undefined ? '' : ` window-type="${options.windowType}"`) +
618+
(options.windowLayer === undefined ? '' : ` window-layer="${options.windowLayer}"`) +
535619
(options.windowActive === undefined ? '' : ` window-active="${options.windowActive}"`);
536620
const open = `<node index="0"${windowAttrs} text="${options.text ?? ''}" resource-id="${options.resourceId ?? ''}" class="${options.className}" package="${options.packageName}" visible-to-user="true" enabled="true" bounds="[0,0][100,100]"`;
537621
return options.children === undefined

‎packages/platform-android/src/__tests__/snapshot.test.ts‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -810,9 +810,7 @@ test('snapshotAndroid keeps helper output when application and system windows ar
810810
appBundleId: 'io.example.fabric',
811811
});
812812

813-
assert.equal(result.androidSnapshot.backend, 'android-helper');
814-
assert.equal(
815-
result.nodes.some((node) => node.label === 'Fabric dashboard'),
816-
true,
817-
);
813+
const { backend, foregroundWindowPackage } = result.androidSnapshot;
814+
assert.deepEqual([backend, foregroundWindowPackage], ['android-helper', 'io.example.fabric']);
815+
assert.ok(result.nodes.some((node) => node.label === 'Fabric dashboard'));
818816
});

‎packages/platform-android/src/snapshot-content-recovery.ts‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { classifyAndroidAlertIdentifier } from './alert-detection.ts';
55
import type { AndroidSnapshotBackendMetadata } from './snapshot-types.ts';
66
import { androidUiNodes, type AndroidUiNodeMetadata } from './ui-hierarchy.ts';
77
import { ANDROID_WINDOW_TYPE_APPLICATION } from './ui-hierarchy-node.ts';
8+
import { isAndroidForegroundWindow } from './ui-hierarchy-visibility.ts';
89

910
const MAX_REPORTED_WINDOW_TYPES = 8;
1011
const MIN_FOREGROUND_APP_MEANINGFUL_NODES = 2;
@@ -44,10 +45,19 @@ export function classifyAndroidHelperContent(
4445
xml: string,
4546
metadata: AndroidSnapshotBackendMetadata,
4647
options: { foregroundAppPackage?: string } = {},
47-
): AndroidHelperContentClassification {
48+
): AndroidHelperContentClassification & { foregroundWindowPackage?: string } {
4849
if (metadata.backend !== 'android-helper') return { outcome: 'ok' };
49-
5050
const summary = summarizeAndroidHelperXml(xml, options.foregroundAppPackage);
51+
const content = classifyAndroidHelperSummary(summary, metadata);
52+
return summary.foregroundWindow
53+
? { ...content, foregroundWindowPackage: summary.foregroundWindow.packageName }
54+
: content;
55+
}
56+
57+
function classifyAndroidHelperSummary(
58+
summary: AndroidHelperXmlSummary,
59+
metadata: AndroidSnapshotBackendMetadata,
60+
): AndroidHelperContentClassification {
5161
if (isEmptyHelperOutput(summary, metadata)) {
5262
return unusable(
5363
summary,
@@ -201,6 +211,11 @@ type AndroidHelperXmlSummary = {
201211
foregroundAppPackage?: string;
202212
foregroundAppMeaningfulNodeCount?: number;
203213
windowTypes: number[];
214+
/**
215+
* The non-system application window the presentation keeps as foreground: active or focused, at
216+
* the highest layer. A stale application window under it is not foreground.
217+
*/
218+
foregroundWindow?: { packageName: string; layer: number };
204219
};
205220

206221
type AndroidHelperXmlSummaryState = Omit<AndroidHelperXmlSummary, 'windowTypes'> & {
@@ -280,9 +295,20 @@ function recordAndroidHelperWindowNode(
280295
if (node.windowType === ANDROID_WINDOW_TYPE_APPLICATION) {
281296
summary.applicationWindowRootCount += 1;
282297
if (summary.currentWindowSystemOwned) summary.systemApplicationWindowRootCount += 1;
298+
else recordAndroidForegroundWindow(summary, node);
283299
}
284300
}
285301

302+
function recordAndroidForegroundWindow(
303+
summary: AndroidHelperXmlSummaryState,
304+
node: AndroidUiNodeMetadata,
305+
): void {
306+
if (node.packageName === null || !isAndroidForegroundWindow(node)) return;
307+
const layer = node.windowLayer ?? -1;
308+
if (summary.foregroundWindow && summary.foregroundWindow.layer >= layer) return;
309+
summary.foregroundWindow = { packageName: node.packageName, layer };
310+
}
311+
286312
function recordAndroidHelperMeaningfulNode(
287313
summary: AndroidHelperXmlSummaryState,
288314
node: AndroidUiNodeMetadata,
@@ -370,6 +396,7 @@ function finalizeAndroidHelperXmlSummary(
370396
}
371397
: {}),
372398
windowTypes: [...summary.windowTypes].sort((a, b) => a - b).slice(0, MAX_REPORTED_WINDOW_TYPES),
399+
...(summary.foregroundWindow ? { foregroundWindow: summary.foregroundWindow } : {}),
373400
};
374401
}
375402

‎packages/platform-android/src/snapshot-types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,11 @@ export type AndroidSnapshotBackendMetadata = {
4141
cacheInvalidated?: boolean;
4242
captureMode?: AndroidSnapshotCaptureMode;
4343
systemSurfaceOnly?: boolean;
44+
/**
45+
* Package of the foreground application window the snapshot helper captured: active or focused,
46+
* at the highest layer, and not owned by the system. Absent when the helper saw none.
47+
*/
48+
foregroundWindowPackage?: string;
4449
windowCount?: number;
4550
/**
4651
* `AccessibilityWindowInfo` types of the listed windows the helper could not serialize because

‎packages/platform-android/src/snapshot.ts‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -639,9 +639,13 @@ async function captureAndroidHelperContentAttempt(params: {
639639
capture: {
640640
xml: helperCapture.xml,
641641
helperMetadata: helperCapture.helperMetadata,
642-
metadata: systemSurfaceOnly
643-
? { ...helperCapture.metadata, systemSurfaceOnly: true }
644-
: helperCapture.metadata,
642+
metadata: {
643+
...helperCapture.metadata,
644+
...(systemSurfaceOnly ? { systemSurfaceOnly: true } : {}),
645+
...(content.foregroundWindowPackage
646+
? { foregroundWindowPackage: content.foregroundWindowPackage }
647+
: {}),
648+
},
645649
},
646650
};
647651
}

‎packages/platform-android/src/ui-hierarchy-visibility.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,9 @@ function isAndroidApplicationWindow(node: AndroidNode): boolean {
9292
return node.windowType === ANDROID_WINDOW_TYPE_APPLICATION;
9393
}
9494

95-
function isAndroidForegroundWindow(node: AndroidNode): boolean {
95+
/** A window the user is in: active or focused. Shared with the snapshot header's foreground app. */
96+
export function isAndroidForegroundWindow(
97+
node: Pick<AndroidNode, 'windowActive' | 'windowFocused'>,
98+
): boolean {
9699
return node.windowActive === true || node.windowFocused === true;
97100
}

‎website/docs/docs/snapshots.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ and disclosure behavior ([#2682](https://github.com/callstack/agent-device/issue
8080
now, and hints that point to hidden list content.
8181
- Use `snapshot -i` by default in agent loops.
8282
- On Android, `snapshot -i` also keeps the title and message of a standard alert dialog, so you can see what its buttons confirm.
83+
- When the Android snapshot helper captures the tree, the `Page:` and `App:` lines (`appName` and `appBundleId` in `--json`) name the foreground application window: active or focused, at the highest layer, and not a system UI or keyboard window. After a failed `open`, for example, they name the launcher. An app other than the session app shows its package name in both lines. `androidSnapshot.foregroundWindowPackage` in `--json` carries it, and the session keeps its app. The UIAutomator fallback names the session app.
8384
- On Android, repeating an unfiltered snapshot when nothing on screen changed (content or bounds) returns a short acknowledgement instead of the tree. `-i`, `-d`, `-s`, `--json`, and `--raw` always print full output. Add `--force-full` to get the full tree anyway.
8485
- The default text output is a compact view for planning and targeting actions. It lists visible elements first and may collapse helper and accessibility noise. Use `--raw` or `--json` when you need the full provider tree. Keep `--raw` for troubleshooting, such as when you need the full off-screen tree that summaries abbreviate.
8586
- Off-screen interactive content is collapsed into summaries that show only a few labels, such as `[off-screen below] 3 interactive items: "Privacy", "Battery", "About"`.

0 commit comments

Comments
 (0)