Skip to content

Commit 80e097e

Browse files
thymikeeclaude
andcommitted
fix(snapshot): name the header app from Android application windows
Address review: scanning node packages let system UI, framework dialogs or an IME win the header when the session app was absent. The Android helper classifier already reads window types, so it now lists the captured non-system application windows (androidSnapshot.applicationWindowPackages), and the header names the topmost one only when none belongs to the session app. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 99ab749 commit 80e097e

8 files changed

Lines changed: 142 additions & 84 deletions

File tree

‎packages/command-runtime/src/commands/capture/runtime/snapshot-app-identity.test.ts‎

Lines changed: 49 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -6,58 +6,58 @@ import { localCommandPolicy } from '@agent-device/contracts/command-policy';
66
import type { CommandSessionStore } from '@agent-device/contracts/runtime-contract';
77
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
88

9-
test('an Android snapshot names the app it captured, not a session app that is not in front', async () => {
10-
const captureWith = async (overlay: { type: string; label: string; bundleId: string }) => {
11-
let stored: Parameters<CommandSessionStore['set']>[0] | undefined;
12-
const device = createAgentDevice({
13-
backend: {
14-
platform: 'android',
15-
captureSnapshot: async () => ({
16-
snapshot: makeSnapshotState(
17-
[
18-
{
19-
index: 0,
20-
depth: 0,
21-
type: 'TextView',
22-
label: '12:00',
23-
bundleId: 'com.android.systemui',
24-
systemBar: 'status',
25-
},
26-
{ index: 1, depth: 0, ...overlay },
27-
],
28-
{ backend: 'android' },
29-
),
30-
appName: 'Settings',
31-
appBundleId: 'com.android.settings',
9+
async function snapshotWithAppWindows(applicationWindowPackages: string[] | undefined) {
10+
let stored: Parameters<CommandSessionStore['set']>[0] | undefined;
11+
const device = createAgentDevice({
12+
backend: {
13+
platform: 'android',
14+
captureSnapshot: async () => ({
15+
snapshot: makeSnapshotState([{ index: 0, depth: 0, type: 'TextView', label: 'Gmail' }], {
16+
backend: 'android',
3217
}),
33-
},
34-
artifacts: createLocalArtifactAdapter(),
35-
sessions: {
36-
get: () => undefined,
37-
set: (record) => {
38-
stored = record;
18+
androidSnapshot: {
19+
backend: 'android-helper',
20+
...(applicationWindowPackages ? { applicationWindowPackages } : {}),
3921
},
22+
appName: 'Settings',
23+
appBundleId: 'com.android.settings',
24+
}),
25+
},
26+
artifacts: createLocalArtifactAdapter(),
27+
sessions: {
28+
get: () => undefined,
29+
set: (record) => {
30+
stored = record;
4031
},
41-
policy: localCommandPolicy(),
42-
});
43-
const result = await device.capture.snapshot({ session: 'default', interactiveOnly: true });
44-
return { result, stored };
45-
};
46-
47-
const launcher = await captureWith({
48-
type: 'ScrollView',
49-
label: 'Workspace',
50-
bundleId: 'com.google.android.apps.nexuslauncher',
32+
},
33+
policy: localCommandPolicy(),
5134
});
52-
assert.equal(launcher.result.appName, 'com.google.android.apps.nexuslauncher');
53-
assert.equal(launcher.result.appBundleId, 'com.google.android.apps.nexuslauncher');
54-
assert.equal(launcher.stored?.appBundleId, 'com.android.settings');
35+
const result = await device.capture.snapshot({ session: 'default', interactiveOnly: true });
36+
return { result, stored };
37+
}
5538

56-
const settings = await captureWith({
57-
type: 'TextView',
58-
label: 'Network & internet',
59-
bundleId: 'com.android.settings',
60-
});
61-
assert.equal(settings.result.appName, 'Settings');
62-
assert.equal(settings.result.appBundleId, 'com.android.settings');
39+
test('an Android snapshot names the captured app when the session app has no window in it', async () => {
40+
const { result, stored } = await snapshotWithAppWindows([
41+
'com.google.android.apps.nexuslauncher',
42+
]);
43+
44+
assert.equal(result.appName, 'com.google.android.apps.nexuslauncher');
45+
assert.equal(result.appBundleId, 'com.google.android.apps.nexuslauncher');
46+
assert.equal(stored?.appBundleId, 'com.android.settings');
47+
});
48+
49+
test('an Android snapshot keeps the session app while one of its windows is captured', async () => {
50+
const { result } = await snapshotWithAppWindows([
51+
'com.google.android.permissioncontroller',
52+
'com.android.settings',
53+
]);
54+
55+
assert.equal(result.appName, 'Settings');
56+
assert.equal(result.appBundleId, 'com.android.settings');
57+
});
58+
59+
test('a snapshot without application window facts names the session app', async () => {
60+
const { result } = await snapshotWithAppWindows(undefined);
61+
62+
assert.equal(result.appBundleId, 'com.android.settings');
6363
});

‎packages/command-runtime/src/commands/capture/runtime/snapshot.ts‎

Lines changed: 8 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,7 @@ import type {
1919
SnapshotVisibility,
2020
} from '@agent-device/kernel/snapshot';
2121
import type { SnapshotState } from '@agent-device/contracts/snapshot-model';
22-
import type {
23-
AgentDeviceBackendPlatform,
24-
BackendSnapshotResult,
25-
} from '@agent-device/contracts/backend';
22+
import type { BackendSnapshotResult } from '@agent-device/contracts/backend';
2623
import type {
2724
AgentDeviceRuntime,
2825
CommandSessionRecord,
@@ -125,7 +122,7 @@ export const snapshotCommand: RuntimeCommand<
125122
...(capture.snapshot.presentedSurface
126123
? { presentedSurface: capture.snapshot.presentedSurface }
127124
: {}),
128-
...snapshotAppFields(capture, runtime.backend.platform),
125+
...snapshotAppFields(capture),
129126
});
130127
};
131128

@@ -238,22 +235,18 @@ function nextSnapshotSession(
238235
}
239236

240237
/**
241-
* The app the response header names. Every Android node carries its window's package, so a capture
242-
* whose app content belongs only to another package, such as the launcher after a failed open,
243-
* names that package instead of the session app. The session keeps its app.
238+
* The app the response header names. An Android helper capture lists its application windows, so
239+
* one that shows none of the session app, such as the launcher after a failed open, names the
240+
* topmost captured app instead. The session keeps its app.
244241
*/
245-
function snapshotAppFields(
246-
capture: SnapshotCapture,
247-
platform: AgentDeviceBackendPlatform,
248-
): {
242+
function snapshotAppFields(capture: SnapshotCapture): {
249243
appName?: string;
250244
appBundleId?: string;
251245
} {
252246
const sessionApp = capture.result.appBundleId ?? capture.session?.appBundleId;
247+
const appWindows = capture.annotations.androidSnapshot?.applicationWindowPackages;
253248
const shownApp =
254-
platform === 'android' && sessionApp
255-
? capturedAndroidApp(capture.snapshot.nodes, sessionApp)
256-
: sessionApp;
249+
sessionApp && appWindows?.[0] && !appWindows.includes(sessionApp) ? appWindows[0] : sessionApp;
257250
const appName =
258251
shownApp === sessionApp ? (capture.result.appName ?? capture.session?.appName) : shownApp;
259252
return {
@@ -262,17 +255,6 @@ function snapshotAppFields(
262255
};
263256
}
264257

265-
/** The session app when any app node belongs to it, else the first other app package. */
266-
function capturedAndroidApp(nodes: readonly SnapshotNode[], sessionApp: string): string {
267-
let otherApp: string | undefined;
268-
for (const node of nodes) {
269-
if (!node.bundleId || node.systemBar || node.keyboardOwned) continue;
270-
if (node.bundleId === sessionApp) return sessionApp;
271-
otherApp ??= node.bundleId;
272-
}
273-
return otherApp ?? sessionApp;
274-
}
275-
276258
/**
277259
* A capture that reported nothing about truncation is only "not truncated" when its producer
278260
* actually observes truncation. Producers that do not (Appium page source, the Limrun element

‎packages/contracts/src/snapshot-types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,11 @@ export type AndroidSnapshotBackendMetadata = {
7979
cacheInvalidated?: boolean;
8080
captureMode?: string;
8181
systemSurfaceOnly?: boolean;
82+
/**
83+
* Packages of the captured application windows that are not system-owned, topmost first. Input
84+
* method and system windows are excluded. Absent when the capture read no such window.
85+
*/
86+
applicationWindowPackages?: string[];
8287
windowCount?: number;
8388
/**
8489
* `AccessibilityWindowInfo` types of the listed windows the helper could not serialize because

‎packages/platform-android/src/__tests__/snapshot-content-recovery.test.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,38 @@ test('rejects system chrome inherited under an empty application window', () =>
270270
expect(decision?.diagnostics.helperNonSystemMeaningfulNodeCount).toBe(0);
271271
});
272272

273+
test('lists only non-system application windows, topmost first', () => {
274+
const window = (windowType: number, packageName: string, text: string) =>
275+
node({
276+
windowType,
277+
packageName,
278+
className: 'android.widget.FrameLayout',
279+
children: [node({ text, packageName, className: 'android.widget.TextView' })],
280+
});
281+
const result = classifyAndroidHelperContent(
282+
helperXml([
283+
window(3, 'com.android.systemui', '12:00'),
284+
window(1, 'android', 'Allow?'),
285+
window(2, 'com.google.android.inputmethod.latin', 'Space'),
286+
window(1, 'com.google.android.apps.nexuslauncher', 'Gmail'),
287+
window(1, 'com.google.android.apps.nexuslauncher', 'Photos'),
288+
window(1, 'com.example.app', 'Home'),
289+
]),
290+
{
291+
backend: 'android-helper',
292+
nodeCount: 12,
293+
rootPresent: true,
294+
windowCount: 6,
295+
captureMode: 'interactive-windows',
296+
},
297+
);
298+
299+
expect(result.outcome === 'unusable' ? [] : result.applicationWindowPackages).toEqual([
300+
'com.google.android.apps.nexuslauncher',
301+
'com.example.app',
302+
]);
303+
});
304+
273305
test('returns a meaningful active system surface (notification shade) faithfully', () => {
274306
const result = classifyAndroidHelperContent(
275307
helperXml([

‎packages/platform-android/src/snapshot-content-recovery.ts‎

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,19 +35,41 @@ export type AndroidHelperContentRecoveryDecision = {
3535
};
3636
};
3737

38-
export type AndroidHelperContentClassification =
38+
type AndroidHelperContentOutcome =
3939
| { outcome: 'ok' }
4040
| { outcome: 'system-surface-only' }
4141
| { outcome: 'unusable'; decision: AndroidHelperContentRecoveryDecision };
4242

43+
export type AndroidHelperContentClassification =
44+
| ({ outcome: 'ok' | 'system-surface-only' } & AndroidApplicationWindows)
45+
| { outcome: 'unusable'; decision: AndroidHelperContentRecoveryDecision };
46+
47+
type AndroidApplicationWindows = {
48+
/**
49+
* Packages of the captured application windows that are not system-owned, topmost first: the
50+
* app a snapshot header names. Input method and system windows are never application windows.
51+
*/
52+
applicationWindowPackages: string[];
53+
};
54+
4355
export function classifyAndroidHelperContent(
4456
xml: string,
4557
metadata: AndroidSnapshotBackendMetadata,
4658
options: { foregroundAppPackage?: string } = {},
4759
): AndroidHelperContentClassification {
48-
if (metadata.backend !== 'android-helper') return { outcome: 'ok' };
49-
60+
if (metadata.backend !== 'android-helper')
61+
return { outcome: 'ok', applicationWindowPackages: [] };
5062
const summary = summarizeAndroidHelperXml(xml, options.foregroundAppPackage);
63+
const content = classifyAndroidHelperSummary(summary, metadata);
64+
return content.outcome === 'unusable'
65+
? content
66+
: { ...content, applicationWindowPackages: summary.applicationWindowPackages };
67+
}
68+
69+
function classifyAndroidHelperSummary(
70+
summary: AndroidHelperXmlSummary,
71+
metadata: AndroidSnapshotBackendMetadata,
72+
): AndroidHelperContentOutcome {
5173
if (isEmptyHelperOutput(summary, metadata)) {
5274
return unusable(
5375
summary,
@@ -97,7 +119,7 @@ function unusable(
97119
metadata: AndroidSnapshotBackendMetadata,
98120
reason: AndroidHelperContentRecoveryDecision['reason'],
99121
failureReason: string,
100-
): AndroidHelperContentClassification {
122+
): AndroidHelperContentOutcome {
101123
return {
102124
outcome: 'unusable',
103125
decision: buildRecoveryDecision(summary, metadata, reason, failureReason),
@@ -201,7 +223,7 @@ type AndroidHelperXmlSummary = {
201223
foregroundAppPackage?: string;
202224
foregroundAppMeaningfulNodeCount?: number;
203225
windowTypes: number[];
204-
};
226+
} & AndroidApplicationWindows;
205227

206228
type AndroidHelperXmlSummaryState = Omit<AndroidHelperXmlSummary, 'windowTypes'> & {
207229
currentWindowType?: number;
@@ -243,6 +265,7 @@ function createAndroidHelperXmlSummaryState(
243265
? { foregroundAppPackage, foregroundAppMeaningfulNodeCount: 0 }
244266
: {}),
245267
windowTypes: new Set<number>(),
268+
applicationWindowPackages: [],
246269
};
247270
}
248271

@@ -280,6 +303,12 @@ function recordAndroidHelperWindowNode(
280303
if (node.windowType === ANDROID_WINDOW_TYPE_APPLICATION) {
281304
summary.applicationWindowRootCount += 1;
282305
if (summary.currentWindowSystemOwned) summary.systemApplicationWindowRootCount += 1;
306+
else if (
307+
node.packageName !== null &&
308+
!summary.applicationWindowPackages.includes(node.packageName)
309+
) {
310+
summary.applicationWindowPackages.push(node.packageName);
311+
}
283312
}
284313
}
285314

@@ -370,6 +399,7 @@ function finalizeAndroidHelperXmlSummary(
370399
}
371400
: {}),
372401
windowTypes: [...summary.windowTypes].sort((a, b) => a - b).slice(0, MAX_REPORTED_WINDOW_TYPES),
402+
applicationWindowPackages: summary.applicationWindowPackages,
373403
};
374404
}
375405

‎packages/platform-android/src/snapshot-types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,11 @@ export type AndroidSnapshotBackendMetadata = {
4141
cacheInvalidated?: boolean;
4242
captureMode?: AndroidSnapshotCaptureMode;
4343
systemSurfaceOnly?: boolean;
44+
/**
45+
* Packages of the captured application windows that are not system-owned, topmost first. Input
46+
* method and system windows are excluded. Absent when the capture read no such window.
47+
*/
48+
applicationWindowPackages?: string[];
4449
windowCount?: number;
4550
/**
4651
* `AccessibilityWindowInfo` types of the listed windows the helper could not serialize because

‎packages/platform-android/src/snapshot.ts‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -639,9 +639,13 @@ async function captureAndroidHelperContentAttempt(params: {
639639
capture: {
640640
xml: helperCapture.xml,
641641
helperMetadata: helperCapture.helperMetadata,
642-
metadata: systemSurfaceOnly
643-
? { ...helperCapture.metadata, systemSurfaceOnly: true }
644-
: helperCapture.metadata,
642+
metadata: {
643+
...helperCapture.metadata,
644+
...(systemSurfaceOnly ? { systemSurfaceOnly: true } : {}),
645+
...(content.applicationWindowPackages.length > 0
646+
? { applicationWindowPackages: content.applicationWindowPackages }
647+
: {}),
648+
},
645649
},
646650
};
647651
}

‎website/docs/docs/snapshots.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ and disclosure behavior ([#2682](https://github.com/callstack/agent-device/issue
8080
now, and hints that point to hidden list content.
8181
- Use `snapshot -i` by default in agent loops.
8282
- On Android, `snapshot -i` also keeps the title and message of a standard alert dialog, so you can see what its buttons confirm.
83-
- On Android, the `Page:` and `App:` lines (`appName` and `appBundleId` in `--json`) name the app the snapshot shows. If the session app is not in front, for example after a failed `open`, they name the app that is, such as the launcher. The session keeps its app.
83+
- On Android, the `Page:` and `App:` lines (`appName` and `appBundleId` in `--json`) name the session app while the snapshot holds one of its windows. When it holds none, for example after a failed `open`, they name the topmost captured application window, such as the launcher; system UI and keyboard windows never count. `androidSnapshot.applicationWindowPackages` in `--json` lists those windows. The session keeps its app.
8484
- On Android, repeating an unfiltered snapshot when nothing on screen changed (content or bounds) returns a short acknowledgement instead of the tree. `-i`, `-d`, `-s`, `--json`, and `--raw` always print full output. Add `--force-full` to get the full tree anyway.
8585
- The default text output is a compact view for planning and targeting actions. It lists visible elements first and may collapse helper and accessibility noise. Use `--raw` or `--json` when you need the full provider tree. Keep `--raw` for troubleshooting, such as when you need the full off-screen tree that summaries abbreviate.
8686
- Off-screen interactive content is collapsed into summaries that show only a few labels, such as `[off-screen below] 3 interactive items: "Privacy", "Battery", "About"`.

0 commit comments

Comments
 (0)