Skip to content

Commit 5eccba6

Browse files
thymikeeclaude
andcommitted
fix(snapshot): order Android app windows by layer and name one without a session app
Address review: the helper lists windows in getWindows() order, which is not z-order, so the header could name an app behind another. Windows now sort by window layer. A session with no app names the topmost captured app instead of printing no header, and the docs describe the list as deduplicated package names. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 80e097e commit 5eccba6

7 files changed

Lines changed: 63 additions & 27 deletions

File tree

‎packages/command-runtime/src/commands/capture/runtime/snapshot-app-identity.test.ts‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,13 @@ import { localCommandPolicy } from '@agent-device/contracts/command-policy';
66
import type { CommandSessionStore } from '@agent-device/contracts/runtime-contract';
77
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
88

9-
async function snapshotWithAppWindows(applicationWindowPackages: string[] | undefined) {
9+
async function snapshotWithAppWindows(
10+
applicationWindowPackages: string[] | undefined,
11+
sessionApp: { appName: string; appBundleId: string } | undefined = {
12+
appName: 'Settings',
13+
appBundleId: 'com.android.settings',
14+
},
15+
) {
1016
let stored: Parameters<CommandSessionStore['set']>[0] | undefined;
1117
const device = createAgentDevice({
1218
backend: {
@@ -19,8 +25,7 @@ async function snapshotWithAppWindows(applicationWindowPackages: string[] | unde
1925
backend: 'android-helper',
2026
...(applicationWindowPackages ? { applicationWindowPackages } : {}),
2127
},
22-
appName: 'Settings',
23-
appBundleId: 'com.android.settings',
28+
...sessionApp,
2429
}),
2530
},
2631
artifacts: createLocalArtifactAdapter(),
@@ -39,6 +44,7 @@ async function snapshotWithAppWindows(applicationWindowPackages: string[] | unde
3944
test('an Android snapshot names the captured app when the session app has no window in it', async () => {
4045
const { result, stored } = await snapshotWithAppWindows([
4146
'com.google.android.apps.nexuslauncher',
47+
'com.example.behind',
4248
]);
4349

4450
assert.equal(result.appName, 'com.google.android.apps.nexuslauncher');
@@ -59,5 +65,16 @@ test('an Android snapshot keeps the session app while one of its windows is capt
5965
test('a snapshot without application window facts names the session app', async () => {
6066
const { result } = await snapshotWithAppWindows(undefined);
6167

68+
assert.equal(result.appName, 'Settings');
6269
assert.equal(result.appBundleId, 'com.android.settings');
6370
});
71+
72+
test('an Android snapshot in a session with no app names the topmost captured app', async () => {
73+
const { result } = await snapshotWithAppWindows(
74+
['com.google.android.apps.nexuslauncher', 'com.example.behind'],
75+
undefined,
76+
);
77+
78+
assert.equal(result.appName, 'com.google.android.apps.nexuslauncher');
79+
assert.equal(result.appBundleId, 'com.google.android.apps.nexuslauncher');
80+
});

‎packages/command-runtime/src/commands/capture/runtime/snapshot.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -236,8 +236,8 @@ function nextSnapshotSession(
236236

237237
/**
238238
* The app the response header names. An Android helper capture lists its application windows, so
239-
* one that shows none of the session app, such as the launcher after a failed open, names the
240-
* topmost captured app instead. The session keeps its app.
239+
* one that shows none of the session app, such as the launcher after a failed open, or a session
240+
* with no app, names the topmost captured app instead. The session keeps its app.
241241
*/
242242
function snapshotAppFields(capture: SnapshotCapture): {
243243
appName?: string;
@@ -246,7 +246,7 @@ function snapshotAppFields(capture: SnapshotCapture): {
246246
const sessionApp = capture.result.appBundleId ?? capture.session?.appBundleId;
247247
const appWindows = capture.annotations.androidSnapshot?.applicationWindowPackages;
248248
const shownApp =
249-
sessionApp && appWindows?.[0] && !appWindows.includes(sessionApp) ? appWindows[0] : sessionApp;
249+
sessionApp && appWindows?.includes(sessionApp) ? sessionApp : (appWindows?.[0] ?? sessionApp);
250250
const appName =
251251
shownApp === sessionApp ? (capture.result.appName ?? capture.session?.appName) : shownApp;
252252
return {

‎packages/contracts/src/snapshot-types.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -80,8 +80,9 @@ export type AndroidSnapshotBackendMetadata = {
8080
captureMode?: string;
8181
systemSurfaceOnly?: boolean;
8282
/**
83-
* Packages of the captured application windows that are not system-owned, topmost first. Input
84-
* method and system windows are excluded. Absent when the capture read no such window.
83+
* Package names of the captured application windows that are not system-owned, by window layer
84+
* (topmost first), each listed once. Input method and system windows are excluded. Absent when
85+
* the capture read no such window.
8586
*/
8687
applicationWindowPackages?: string[];
8788
windowCount?: number;

‎packages/platform-android/src/__tests__/snapshot-content-recovery.test.ts‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -270,10 +270,11 @@ test('rejects system chrome inherited under an empty application window', () =>
270270
expect(decision?.diagnostics.helperNonSystemMeaningfulNodeCount).toBe(0);
271271
});
272272

273-
test('lists only non-system application windows, topmost first', () => {
274-
const window = (windowType: number, packageName: string, text: string) =>
273+
test('lists only non-system application windows, by layer, topmost first', () => {
274+
const window = (windowType: number, packageName: string, text: string, windowLayer = 0) =>
275275
node({
276276
windowType,
277+
windowLayer,
277278
packageName,
278279
className: 'android.widget.FrameLayout',
279280
children: [node({ text, packageName, className: 'android.widget.TextView' })],
@@ -283,9 +284,10 @@ test('lists only non-system application windows, topmost first', () => {
283284
window(3, 'com.android.systemui', '12:00'),
284285
window(1, 'android', 'Allow?'),
285286
window(2, 'com.google.android.inputmethod.latin', 'Space'),
286-
window(1, 'com.google.android.apps.nexuslauncher', 'Gmail'),
287-
window(1, 'com.google.android.apps.nexuslauncher', 'Photos'),
288-
window(1, 'com.example.app', 'Home'),
287+
// Listed before the app, but below it: getWindows() order is not z-order.
288+
window(1, 'com.google.android.apps.nexuslauncher', 'Gmail', 1),
289+
window(1, 'com.google.android.apps.nexuslauncher', 'Photos', 1),
290+
window(1, 'com.example.app', 'Home', 5),
289291
]),
290292
{
291293
backend: 'android-helper',
@@ -297,8 +299,8 @@ test('lists only non-system application windows, topmost first', () => {
297299
);
298300

299301
expect(result.outcome === 'unusable' ? [] : result.applicationWindowPackages).toEqual([
300-
'com.google.android.apps.nexuslauncher',
301302
'com.example.app',
303+
'com.google.android.apps.nexuslauncher',
302304
]);
303305
});
304306

@@ -558,12 +560,14 @@ function node(options: {
558560
packageName: string;
559561
className: string;
560562
windowType?: number;
563+
windowLayer?: number;
561564
windowActive?: boolean;
562565
/** Nested children, so chrome leaves can sit inside their status/nav-bar container as on a device. */
563566
children?: string[];
564567
}): string {
565568
const windowAttrs =
566569
(options.windowType === undefined ? '' : ` window-type="${options.windowType}"`) +
570+
(options.windowLayer === undefined ? '' : ` window-layer="${options.windowLayer}"`) +
567571
(options.windowActive === undefined ? '' : ` window-active="${options.windowActive}"`);
568572
const open = `<node index="0"${windowAttrs} text="${options.text ?? ''}" resource-id="${options.resourceId ?? ''}" class="${options.className}" package="${options.packageName}" visible-to-user="true" enabled="true" bounds="[0,0][100,100]"`;
569573
return options.children === undefined

‎packages/platform-android/src/snapshot-content-recovery.ts‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,9 @@ export type AndroidHelperContentClassification =
4646

4747
type AndroidApplicationWindows = {
4848
/**
49-
* Packages of the captured application windows that are not system-owned, topmost first: the
50-
* app a snapshot header names. Input method and system windows are never application windows.
49+
* Package names of the captured application windows that are not system-owned, by window layer
50+
* (topmost first), each listed once: the app a snapshot header names. Input method and system
51+
* windows are never application windows.
5152
*/
5253
applicationWindowPackages: string[];
5354
};
@@ -225,7 +226,11 @@ type AndroidHelperXmlSummary = {
225226
windowTypes: number[];
226227
} & AndroidApplicationWindows;
227228

228-
type AndroidHelperXmlSummaryState = Omit<AndroidHelperXmlSummary, 'windowTypes'> & {
229+
type AndroidHelperXmlSummaryState = Omit<
230+
AndroidHelperXmlSummary,
231+
'windowTypes' | 'applicationWindowPackages'
232+
> & {
233+
applicationWindows: Array<{ packageName: string; layer: number }>;
229234
currentWindowType?: number;
230235
currentWindowActiveOrFocused?: boolean;
231236
currentWindowSystemOwned?: boolean;
@@ -265,7 +270,7 @@ function createAndroidHelperXmlSummaryState(
265270
? { foregroundAppPackage, foregroundAppMeaningfulNodeCount: 0 }
266271
: {}),
267272
windowTypes: new Set<number>(),
268-
applicationWindowPackages: [],
273+
applicationWindows: [],
269274
};
270275
}
271276

@@ -303,11 +308,11 @@ function recordAndroidHelperWindowNode(
303308
if (node.windowType === ANDROID_WINDOW_TYPE_APPLICATION) {
304309
summary.applicationWindowRootCount += 1;
305310
if (summary.currentWindowSystemOwned) summary.systemApplicationWindowRootCount += 1;
306-
else if (
307-
node.packageName !== null &&
308-
!summary.applicationWindowPackages.includes(node.packageName)
309-
) {
310-
summary.applicationWindowPackages.push(node.packageName);
311+
else if (node.packageName !== null) {
312+
summary.applicationWindows.push({
313+
packageName: node.packageName,
314+
layer: node.windowLayer ?? Number.NEGATIVE_INFINITY,
315+
});
311316
}
312317
}
313318
}
@@ -399,7 +404,15 @@ function finalizeAndroidHelperXmlSummary(
399404
}
400405
: {}),
401406
windowTypes: [...summary.windowTypes].sort((a, b) => a - b).slice(0, MAX_REPORTED_WINDOW_TYPES),
402-
applicationWindowPackages: summary.applicationWindowPackages,
407+
// The helper lists windows in getWindows() order, which is not z-order; the layer is.
408+
applicationWindowPackages: [
409+
...new Set(
410+
summary.applicationWindows
411+
.slice()
412+
.sort((a, b) => b.layer - a.layer || 0)
413+
.map((window) => window.packageName),
414+
),
415+
],
403416
};
404417
}
405418

‎packages/platform-android/src/snapshot-types.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,9 @@ export type AndroidSnapshotBackendMetadata = {
4242
captureMode?: AndroidSnapshotCaptureMode;
4343
systemSurfaceOnly?: boolean;
4444
/**
45-
* Packages of the captured application windows that are not system-owned, topmost first. Input
46-
* method and system windows are excluded. Absent when the capture read no such window.
45+
* Package names of the captured application windows that are not system-owned, by window layer
46+
* (topmost first), each listed once. Input method and system windows are excluded. Absent when
47+
* the capture read no such window.
4748
*/
4849
applicationWindowPackages?: string[];
4950
windowCount?: number;

‎website/docs/docs/snapshots.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ and disclosure behavior ([#2682](https://github.com/callstack/agent-device/issue
8080
now, and hints that point to hidden list content.
8181
- Use `snapshot -i` by default in agent loops.
8282
- On Android, `snapshot -i` also keeps the title and message of a standard alert dialog, so you can see what its buttons confirm.
83-
- On Android, the `Page:` and `App:` lines (`appName` and `appBundleId` in `--json`) name the session app while the snapshot holds one of its windows. When it holds none, for example after a failed `open`, they name the topmost captured application window, such as the launcher; system UI and keyboard windows never count. `androidSnapshot.applicationWindowPackages` in `--json` lists those windows. The session keeps its app.
83+
- On Android, the `Page:` and `App:` lines (`appName` and `appBundleId` in `--json`) name the session app while the snapshot holds one of its windows. When it holds none, for example after a failed `open`, or the session has no app, they name the app of the topmost captured application window, such as the launcher; system UI and keyboard windows never count. `androidSnapshot.applicationWindowPackages` in `--json` lists those package names, topmost first. The session keeps its app.
8484
- On Android, repeating an unfiltered snapshot when nothing on screen changed (content or bounds) returns a short acknowledgement instead of the tree. `-i`, `-d`, `-s`, `--json`, and `--raw` always print full output. Add `--force-full` to get the full tree anyway.
8585
- The default text output is a compact view for planning and targeting actions. It lists visible elements first and may collapse helper and accessibility noise. Use `--raw` or `--json` when you need the full provider tree. Keep `--raw` for troubleshooting, such as when you need the full off-screen tree that summaries abbreviate.
8686
- Off-screen interactive content is collapsed into summaries that show only a few labels, such as `[off-screen below] 3 interactive items: "Privacy", "Battery", "About"`.

0 commit comments

Comments
 (0)