Skip to content

Commit 80470a0

Browse files
committed
feat(daemon): disclose a foreground repair on every capture-consuming command
The runner now says when it had to re-activate the session app (#2682, PR1). This carries that fact through the daemon so the command that paid the cost reports it: - `IosTargetActivation` moves to the kernel, where `SnapshotState` can carry it, and the capture annotations projection publishes it beside `warnings`. - `src/daemon/system-surface-disclosure.ts` becomes `capture-disclosure.ts`: the surface disclosure plus `withTargetActivationDisclosure`, always appended, never replacing an earlier warning. - Selector, wait, find, snapshot, and interaction routes disclose against the capture *they* consumed. A route with no capture of its own stays silent, so an older tree's fact is never attributed to a later command. - No response level may drop it: `--level digest` keeps `targetActivation`. - `appstate` on Apple now refuses honestly and names the per-command disclosure; help and `docs/commands.md` say the same thing.
1 parent 58f5634 commit 80470a0

23 files changed

Lines changed: 523 additions & 100 deletions

‎.device-evidence/CHECKLIST.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,28 @@ actually called. Assert the pair on the same device run:
7676
`press <x> <y>` and `press @ref` answered from a live ref frame consume no capture, so they carry no
7777
disclosure even though the runner may have re-activated the app to serve them. Treat a silent
7878
interaction as "unknown", never as "no repair". Follow-up: callstack/agent-device#2694.
79+
80+
## PR2 simulator lane — daemon disclosure (apex-2682-proto, iOS 26.2)
81+
82+
Driven with `bin/agent-device.mjs` after `pnpm build && pnpm clean:daemon`, session `apex2682-pr3`,
83+
off-app handoff via `xcrun simctl openurl <UDID> https://example.com` before each command. Every
84+
command below re-activated the session app, and each answered with `targetActivation` plus the
85+
appended warning:
86+
87+
| Command | Result |
88+
| --- | --- |
89+
| `snapshot -i` | warning first in `warnings`, ahead of the AX-backend warning; `targetActivation` carried |
90+
| `snapshot -i --level digest` | `targetActivation` survives the digest projection |
91+
| `find "Tab Bar" --first` | disclosed on the matched response |
92+
| `press 'label="INFO"'` | disclosed on the interaction response |
93+
| `wait 'label="INFO"' 8000` | disclosed on the satisfied response |
94+
95+
`otherActiveApplicationPid` on the simulator was the Safari process, consistent across commands (pid 33878).
96+
97+
### Proof the review asked for on this lane
98+
99+
The false-attribution class is covered by unit tests rather than a device run, because the device
100+
cannot show a cache hit on command: `src/daemon/__tests__/capture-disclosure-target-activation.test.ts`
101+
pins that a cache-reused tree discloses nothing, and
102+
`src/daemon/interaction/internal/__tests__/interaction-target-activation-disclosure.test.ts` pins that
103+
a press which consumed no capture is not disclosed against an older tree.

‎packages/capture-kit/src/snapshot-state.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import {
88
attachRefs,
99
buildSnapshotPresentationKey,
1010
snapshotPresentationOptionsFromFlags,
11+
type IosTargetActivation,
1112
type RawSnapshotNode,
1213
type SnapshotBackend,
1314
type SnapshotCaptureProvenance,
@@ -46,6 +47,8 @@ export function buildSnapshotState(
4647
quality?: unknown;
4748
comparisonIdentity?: IosSnapshotComparisonIdentity;
4849
systemSurface?: IosSystemSurfaceProvenance;
50+
/** Foreground repair this capture's own command performed (#2682). */
51+
targetActivation?: IosTargetActivation;
4952
/** The keyboard band the producer measured, carried to the state the tap guards read (#2660). */
5053
keyboard?: SnapshotKeyboardBandFact;
5154
} & SnapshotCaptureProvenance,
@@ -84,6 +87,7 @@ export function buildSnapshotState(
8487
? { comparisonKey: iosSnapshotComparisonIdentityKey(data.comparisonIdentity) }
8588
: {}),
8689
...(data.systemSurface ? { iosSystemSurfaceBundleId: data.systemSurface.bundleId } : {}),
90+
...(data.targetActivation ? { targetActivation: data.targetActivation } : {}),
8791
...(data.keyboard ? { keyboard: data.keyboard } : {}),
8892
presentationKey: buildSnapshotPresentationKey(snapshotPresentationOptionsFromFlags(flags)),
8993
// Only broad Android snapshots become freshness baselines. If the user asked for a scoped

‎packages/contracts/src/ios-target-activation.ts‎

Lines changed: 14 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,28 @@
11
/**
22
* The Apple runner repairs foreground loss by activating the session app before answering a
33
* non-lifecycle command (#2682). That decides which app the answer describes, so the runner stamps
4-
* the repair on the response of the command that paid for it and this module owns what it means to
5-
* an agent. `@agent-device/platform-apple`'s runner reader is the single decoder of the wire shape;
6-
* nothing else parses it.
4+
* the repair on the response of the command that paid for it. `@agent-device/kernel/snapshot` owns
5+
* the fact — it travels on a captured tree's state — and `@agent-device/platform-apple`'s runner
6+
* reader is the single decoder of the wire shape. This module owns only what the fact says to an
7+
* agent, so the sentence cannot drift between the capture path and the daemon routes.
78
*
89
* The pid it carries is a liveness claim, not a foreground owner: the private AX client the runner
910
* already uses for process matching exposes no ordering of `activeApplications`, resolves pids only,
1011
* and the escalation that would name an arbitrary app (`proc_pidpath`) has no iOS SDK declaration and
1112
* no physical-device evidence. So the fact states which other application was alive and leaves the
1213
* reader to conclude no more than that.
1314
*/
15+
import type { IosTargetActivation } from '@agent-device/kernel/snapshot';
1416

15-
/** Reasons the runner can stamp; mirrors its `activateTarget(bundleId:reason:)` call sites. */
16-
export const IOS_TARGET_ACTIVATION_REASONS = [
17-
'bundle_changed',
18-
'stale_target',
19-
'missing_after_wait',
20-
'interaction_foreground_guard',
21-
] as const;
22-
23-
export type IosTargetActivationReason = (typeof IOS_TARGET_ACTIVATION_REASONS)[number];
24-
25-
/**
26-
* States an activation could have been needed for. `runningForeground` is excluded because the
27-
* runner skips `activate()` when the app is already foreground and never stamps a fact there.
28-
*/
29-
export const IOS_TARGET_ACTIVATION_PRIOR_STATES = [
30-
'unknown',
31-
'notRunning',
32-
'runningBackground',
33-
'runningBackgroundSuspended',
34-
] as const;
35-
36-
export type IosTargetActivationPriorState = (typeof IOS_TARGET_ACTIVATION_PRIOR_STATES)[number];
37-
38-
/**
39-
* Foreground repair performed while serving one command. `priorState` is the session app's state
40-
* BEFORE the runner activated it — read before `activate()` ran — so the fact describes what was
41-
* repaired rather than what the repair produced, and it can never report `runningForeground`.
42-
* `otherActiveApplicationPid` is present only when exactly one application other than the session app
43-
* held an active accessibility session at that moment: proof that it was alive, not proof that it
44-
* owned the screen.
45-
*/
46-
export type IosTargetActivation = Readonly<{
47-
reason: IosTargetActivationReason;
48-
priorState: IosTargetActivationPriorState;
49-
otherActiveApplicationPid?: number;
50-
}>;
17+
export type {
18+
IosTargetActivation,
19+
IosTargetActivationPriorState,
20+
IosTargetActivationReason,
21+
} from '@agent-device/kernel/snapshot';
22+
export {
23+
IOS_TARGET_ACTIVATION_PRIOR_STATES,
24+
IOS_TARGET_ACTIVATION_REASONS,
25+
} from '@agent-device/kernel/snapshot';
5126

5227
/**
5328
* What the earlier captures showed. Naming the live pid is a pointer to the only candidate, never a

‎packages/contracts/src/snapshot-capture-annotations.ts‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import type { SnapshotQualityVerdict } from '@agent-device/kernel/snapshot';
1+
import type { IosTargetActivation, SnapshotQualityVerdict } from '@agent-device/kernel/snapshot';
22
import type { AndroidSnapshotBackendMetadata } from './snapshot-types.ts';
33

44
export type SnapshotCaptureAnalysis = {
@@ -19,11 +19,13 @@ export type SnapshotCaptureAnnotations = {
1919
freshness?: SnapshotCaptureFreshness;
2020
quality?: SnapshotQualityVerdict;
2121
warnings?: string[];
22+
/** The Apple runner re-activated the session app while serving this capture (#2682). */
23+
targetActivation?: IosTargetActivation;
2224
};
2325

2426
export type PublicSnapshotCaptureAnnotations = Pick<
2527
SnapshotCaptureAnnotations,
26-
'androidSnapshot' | 'warnings'
28+
'androidSnapshot' | 'warnings' | 'targetActivation'
2729
> & {
2830
snapshotQuality?: SnapshotQualityVerdict;
2931
};
@@ -38,6 +40,7 @@ export function snapshotCaptureAnnotationsFrom(
3840
...(source.freshness ? { freshness: source.freshness } : {}),
3941
...(quality ? { quality } : {}),
4042
...(source.warnings ? { warnings: source.warnings } : {}),
43+
...(source.targetActivation ? { targetActivation: source.targetActivation } : {}),
4144
};
4245
}
4346

@@ -50,6 +53,7 @@ export function publicSnapshotCaptureAnnotations(
5053
...(annotations.warnings && annotations.warnings.length > 0
5154
? { warnings: annotations.warnings }
5255
: {}),
56+
...(annotations.targetActivation ? { targetActivation: annotations.targetActivation } : {}),
5357
};
5458
}
5559

@@ -64,15 +68,26 @@ export function readSerializedSnapshotCaptureAnnotations(
6468
? data.warnings.filter((entry): entry is string => typeof entry === 'string')
6569
: undefined;
6670
const quality = readSnapshotQualityVerdict(data.snapshotQuality);
71+
const targetActivation = readTargetActivation(data.targetActivation);
6772
return publicSnapshotCaptureAnnotations({
6873
...(androidSnapshot
6974
? { androidSnapshot: androidSnapshot as AndroidSnapshotBackendMetadata }
7075
: {}),
7176
...(quality ? { quality } : {}),
7277
...(warnings ? { warnings } : {}),
78+
...(targetActivation ? { targetActivation } : {}),
7379
});
7480
}
7581

82+
/** Re-read of a fact this module projected; the declared keys are the only ones it publishes. */
83+
function readTargetActivation(value: unknown): IosTargetActivation | undefined {
84+
if (typeof value !== 'object' || value === null || Array.isArray(value)) return undefined;
85+
const raw = value as Record<string, unknown>;
86+
return typeof raw.reason === 'string' && typeof raw.priorState === 'string'
87+
? (raw as unknown as IosTargetActivation)
88+
: undefined;
89+
}
90+
7691
function readSnapshotQualityVerdict(value: unknown): SnapshotQualityVerdict | undefined {
7792
if (!value || typeof value !== 'object') return undefined;
7893
const raw = value as Record<string, unknown>;

‎packages/kernel/src/snapshot.ts‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,46 @@ export function usesMobileSnapshotPresentation(backend: SnapshotBackend | undefi
415415
);
416416
}
417417

418+
/**
419+
* Reasons the Apple runner can stamp when serving a command required re-activating the session app
420+
* (#2682). Mirrors its `activateTarget(bundleId:reason:)` call sites.
421+
*/
422+
export const IOS_TARGET_ACTIVATION_REASONS = [
423+
'bundle_changed',
424+
'stale_target',
425+
'missing_after_wait',
426+
'interaction_foreground_guard',
427+
] as const;
428+
429+
export type IosTargetActivationReason = (typeof IOS_TARGET_ACTIVATION_REASONS)[number];
430+
431+
/**
432+
* States an activation could have been needed for, in `XCApplicationState` raw order.
433+
* `runningForeground` is excluded because the runner skips `activate()` when the app is already
434+
* foreground and never stamps a fact there.
435+
*/
436+
export const IOS_TARGET_ACTIVATION_PRIOR_STATES = [
437+
'unknown',
438+
'notRunning',
439+
'runningBackground',
440+
'runningBackgroundSuspended',
441+
] as const;
442+
443+
export type IosTargetActivationPriorState = (typeof IOS_TARGET_ACTIVATION_PRIOR_STATES)[number];
444+
445+
/**
446+
* Foreground repair the Apple runner performed while serving one command (#2682). `priorState` is
447+
* the session app's state BEFORE the runner activated it, so the fact describes what was repaired
448+
* rather than what the repair produced. `foregroundPid` is present only when exactly one
449+
* application other than the session app had an active accessibility session at that moment: the
450+
* private AX client the runner reads resolves pids and answers no bundle id for an arbitrary app.
451+
*/
452+
export type IosTargetActivation = Readonly<{
453+
reason: IosTargetActivationReason;
454+
priorState: IosTargetActivationPriorState;
455+
foregroundPid?: number;
456+
}>;
457+
418458
export type SnapshotState = {
419459
nodes: SnapshotNode[];
420460
createdAt: number;
@@ -443,6 +483,12 @@ export type SnapshotState = {
443483
* needs no geometry to be plausible (#2660). Absent means the guard measures the tree as before.
444484
*/
445485
keyboard?: SnapshotKeyboardBandFact;
486+
/**
487+
* iOS: this capture's own command found the session app out of foreground and the runner
488+
* activated it before answering, so an earlier observation in the session described whatever held
489+
* the foreground instead (#2682). Consumers that surface this tree disclose the repair.
490+
*/
491+
targetActivation?: IosTargetActivation;
446492
} & SnapshotStateProvenance;
447493

448494
export type SnapshotUnchanged = {

‎packages/platform-apple/src/runtime.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ const focusKindUnavailable = Object.freeze({
103103
const appStateUnavailable = Object.freeze({
104104
available: false,
105105
reason: 'unsupported-platform-leaf',
106-
hint: 'Apple appstate reads the active session state; a sessionless runtime foreground probe is unavailable.',
106+
hint: 'Apple appstate reads the active session state; a sessionless runtime foreground probe is unavailable. Whether the session app actually held the foreground is answered per command instead: a capture whose command had to re-activate it carries a targetActivation disclosure (#2682).',
107107
} as const);
108108
const headlessUnavailable = Object.freeze({
109109
available: false,

‎src/commands/capture/output.test.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,3 +94,36 @@ test('snapshot output presents the materialized fallback screenshot path', async
9494
/Captured a screenshot of this screen automatically as visual truth: \/client\/artifacts\/snapshot-fallback\.png/,
9595
);
9696
});
97+
98+
/**
99+
* The rendered surface an agent actually reads after the runner repaired foreground (#2682): the
100+
* disclosure is printed, and the structured fact survives the JSON projection so a scripted caller
101+
* never has to parse the sentence.
102+
*/
103+
test('a repaired capture renders the disclosure and keeps the structured fact', async () => {
104+
const output = await snapshotCliOutput({
105+
result: {
106+
...buildResult(REPEATED_CHAIN),
107+
warnings: [
108+
'The session app was not foreground when this command arrived (another app (pid 4562) held ' +
109+
'it, prior state runningBackground), so the runner activated it before answering (reason ' +
110+
'stale_target). Any capture taken earlier in this session described another app (pid 4562), ' +
111+
'not the session app. Re-capture now that the session app answers, or drive the other app ' +
112+
'in its own session.',
113+
],
114+
targetActivation: {
115+
reason: 'stale_target',
116+
priorState: 'runningBackground',
117+
foregroundPid: 4562,
118+
},
119+
} as CaptureSnapshotResult,
120+
});
121+
122+
assert.match(String(output.text), /prior state runningBackground/);
123+
assert.match(String(output.text), /drive the other app in its own session/);
124+
assert.deepEqual((output.jsonData as { targetActivation?: unknown }).targetActivation, {
125+
reason: 'stale_target',
126+
priorState: 'runningBackground',
127+
foregroundPid: 4562,
128+
});
129+
});

‎src/commands/system/index.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,8 @@ const TV_REMOTE_LONGPRESS_PRESET_MS = 500;
4040
const CLIPBOARD_ACTION_VALUES = ['read', 'write'] as const;
4141
const KEYBOARD_METADATA_ACTION_VALUES = ['status', 'dismiss', 'enter', 'return'] as const;
4242

43-
const appStateCommandDescription = 'Show foreground app/activity';
43+
const appStateCommandDescription =
44+
'Show foreground app/activity (Android; iOS answers per command)';
4445
const backCommandDescription =
4546
'Navigate back in the app or through system navigation. Use in-app for the app navigation stack and system when the platform back behavior is required.';
4647
const homeCommandDescription =
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import { iosTargetActivationDisclosure } from '@agent-device/contracts/ios-target-activation';
4+
import type { IosTargetActivation } from '@agent-device/kernel/snapshot';
5+
import { withCaptureDisclosures, withTargetActivationDisclosure } from '../capture-disclosure.ts';
6+
import type { DaemonResponse } from '../daemon-request.ts';
7+
8+
const FACT: IosTargetActivation = {
9+
reason: 'stale_target',
10+
priorState: 'runningBackground',
11+
foregroundPid: 4562,
12+
};
13+
14+
function okResponse(data: Record<string, unknown>): DaemonResponse {
15+
return { ok: true, data };
16+
}
17+
18+
function dataOf(response: DaemonResponse): Record<string, unknown> {
19+
assert.ok(response.ok);
20+
assert.ok(response.data);
21+
return response.data;
22+
}
23+
24+
test('a consumed repair is disclosed on the response without replacing earlier warnings', () => {
25+
const response = withTargetActivationDisclosure(
26+
okResponse({ warnings: ['Refs are stale'], targetActivation: undefined }),
27+
{ targetActivation: FACT },
28+
);
29+
const data = dataOf(response);
30+
assert.deepEqual(data.warnings, ['Refs are stale', iosTargetActivationDisclosure(FACT)]);
31+
assert.deepEqual(data.targetActivation, FACT);
32+
});
33+
34+
test('a capture that already disclosed the identical repair is not told twice', () => {
35+
const already = okResponse({ warnings: [iosTargetActivationDisclosure(FACT)] });
36+
const response = withTargetActivationDisclosure(already, { targetActivation: FACT });
37+
assert.deepEqual(dataOf(response).warnings, [iosTargetActivationDisclosure(FACT)]);
38+
});
39+
40+
test('a failure response appends the repair to its hint and keeps the original details', () => {
41+
const response = withTargetActivationDisclosure(
42+
{
43+
ok: false,
44+
error: {
45+
code: 'COMMAND_FAILED',
46+
message: 'selector missed',
47+
details: { blockedBy: 'android_foreground_surface' },
48+
},
49+
},
50+
{ targetActivation: FACT },
51+
);
52+
assert.equal(response.ok, false);
53+
if (response.ok) return;
54+
assert.equal(response.error.message, 'selector missed');
55+
assert.equal(response.error.code, 'COMMAND_FAILED');
56+
assert.equal(response.error.details?.blockedBy, 'android_foreground_surface');
57+
assert.match(String(response.error.details?.hint), /prior state runningBackground/);
58+
});
59+
60+
test('a capture with no repair leaves the response byte-identical', () => {
61+
const untouched = okResponse({ nodes: [], warnings: ['Refs are stale'] });
62+
const response = withTargetActivationDisclosure(untouched, { targetActivation: undefined });
63+
assert.deepEqual(response, untouched);
64+
});
65+
66+
/**
67+
* The two carriers are pre-existing: the surface disclosure extends `data.warning`, the capture's
68+
* own warnings arrive as `data.warnings` (#2438 predates the array). A response can therefore carry
69+
* both, and each disclosure must survive in the carrier it belongs to.
70+
*/
71+
test('surface and foreground disclosures ride one response together', () => {
72+
const response = withCaptureDisclosures(okResponse({ nodes: [] }), {
73+
iosSystemSurfaceBundleId: 'com.apple.SafariViewService',
74+
targetActivation: FACT,
75+
});
76+
const data = dataOf(response);
77+
assert.match(String(data.warning), /system web sign-in sheet/);
78+
assert.deepEqual(data.warnings, [iosTargetActivationDisclosure(FACT)]);
79+
assert.deepEqual(data.targetActivation, FACT);
80+
});

src/daemon/__tests__/system-surface-disclosure.test.ts renamed to src/daemon/__tests__/capture-disclosure.test.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ vi.mock('../device-ready.ts', () => ({
3535

3636
import { resolveTargetDevice } from '@agent-device/device-selection/dispatch-resolve';
3737
import { ANDROID_EMULATOR } from '../../__tests__/test-utils/device-fixtures.ts';
38-
import { withSystemSurfaceDisclosure } from '../system-surface-disclosure.ts';
38+
import { withSystemSurfaceDisclosure } from '../capture-disclosure.ts';
3939

4040
// The occluding-shade capture every scenario below consumes: no application window content, one
4141
// active quick-settings surface. The Android capture route stamps systemSurfaceOnly on both the

0 commit comments

Comments
 (0)