Skip to content

Commit 168af66

Browse files
thymikeeclaude
andcommitted
feat(macos): opt-in native app backend beside XCTest
`AGENT_DEVICE_MACOS_APP_BACKEND=native` routes macOS app sessions to the macOS helper's accessibility actions instead of the XCTest runner. The surface routing takes the backend explicitly, the native interactor refuses runner-only commands with a typed reason rather than starting XCTest, and helper refusals surface as UNSUPPORTED_OPERATION. ADR 0031 records the decision; docs cover the env vars and limits. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent c0fabe6 commit 168af66

27 files changed

Lines changed: 889 additions & 70 deletions
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# ADR 0031: macOS Native App Backend — Accessibility Actions Beside XCTest
2+
3+
## Status
4+
5+
Accepted (2026-10-03). Opt-in; XCTest stays the default app-session backend.
6+
7+
## Rules at a glance
8+
9+
1. `AGENT_DEVICE_MACOS_APP_BACKEND` selects the backend for macOS `app` sessions: `xctest`
10+
(default) or `native`. It is a daemon setting read through `readMacOsAppBackend`: once when the
11+
Apple runtime owner is created, and once per Apple interactor. An unknown value is
12+
`INVALID_ARGS`, never a silent fallback.
13+
2. `macOsSurfaceBackend(surface, appBackend)` in `packages/contracts/src/session-surface.ts` is the
14+
one routing decision. With `native`, the `app` surface is helper-routed exactly like
15+
`frontmost-app`, `desktop`, and `menubar`.
16+
3. A native daemon never starts XCTest on macOS. `macOsNativeBackendFacts` refuses every macOS
17+
operation only the runner serves (recording, `prepare`, `back`, press and hold, gestures and
18+
their viewport) at admission, for every macOS session, with `UNSUPPORTED_OPERATION` and
19+
`reason: 'unsupported-device-backend'`, the reason the physical-iOS XCTest backend uses.
20+
`macOsNativeAppInteractor` is assembled member by member, so no runner-backed member reaches
21+
it; an action that names no app is refused rather than handed to the runner.
22+
4. Pointer actions are accessibility actions only: `AXPress`, focus, value, selected text, or a
23+
scroll bar value. Press and hold is refused at admission; double, secondary, and middle clicks
24+
are refused by the interactor before the helper runs. A press or fill with no pressable
25+
element, or a scroll with no settable scroll bar, is refused by the helper and carries its
26+
`helperReason` under the same `reason`; that vocabulary is pinned by
27+
`contracts/fixtures/macos-native-helper-outcomes.json`. Only keyboard text falls back to events
28+
posted to the app's process.
29+
5. Snapshots of a Chromium-based session app (one shipping `chrome_100_percent.pak`) enable its
30+
accessibility tree (`AXManualAccessibility`, else `AXEnhancedUserInterface`) before traversal.
31+
Only the snapshot that turns the tree on waits for it (up to 1 s); a tree that does not
32+
populate adds a snapshot warning. The tree stays on for the app's lifetime, as for any
33+
assistive client. The app surface walks up to 48 levels deep and reports a deeper tree as
34+
truncated; other helper surfaces keep 12.
35+
6. Screenshots capture the session app's front window by itself through ScreenCaptureKit.
36+
37+
## Context
38+
39+
The XCTest runner drives a macOS app through XCUIApplication, which puts the host in Automation
40+
Mode: a system overlay is shown and the runner moves the shared pointer. The macOS helper already
41+
read the accessibility tree for helper surfaces, and the session snapshot vocabulary is the same,
42+
so an app session can be served without a test session while the app stays behind the user's
43+
windows.
44+
45+
## Decision details
46+
47+
**Pointer delivery.** Events posted to a process with `CGEventPostToPid` were measured against
48+
Calculator (SwiftUI): mouse events were dropped whether the app was frontmost or in the background,
49+
and wheel events were dropped in the background, while every post reported success. Keyboard events
50+
were accepted by Calculator and by Electron apps in the background. A fallback that cannot be
51+
observed to work would turn "nothing happened" into success, so pointer actions have no event
52+
fallback.
53+
54+
**Target resolution.** The helper hit-tests inside the session app (other apps' windows above it do
55+
not answer) and walks at most four ancestors for a text input or a pressable control role.
56+
Chromium answers a hit test with wrapper groups that all claim `AXPress`, so when the chain names
57+
no control the helper picks the smallest such element whose frame contains the point, searching
58+
only the window the hit landed in (the app's front on-screen window when the hit names none). A
59+
group is pressed only when no control contains the point. Responses name the window acted in
60+
(`windowTitle`). This resolves the same point the daemon computed from the snapshot node; the
61+
daemon dispatch paths and their ADR 0011 guarantees are unchanged.
62+
63+
**Pointer dispatch, not element identity.** The daemon dispatches every platform by point, and
64+
the occlusion, offscreen, and parent-owned touch-point guarantees of ADR 0011 are decided on that
65+
point. Sending an element identity instead would be a new dispatch path with its own guarantee
66+
row, and an `AXUIElement` cannot outlive the one-shot helper process that resolved it, so an
67+
identity would be a tree path re-resolved against a tree that may have changed.
68+
69+
## Rejected alternatives
70+
71+
- **Suppressing Automation Mode.** `automationmodetool` removes the authentication prompt, not the
72+
overlay, and the runner still owns the pointer.
73+
- **Private SkyLight event delivery** (`SLEventPostToPid`, focus-without-raise). It would cover
74+
pointer-only controls, but it is private API that can break with any macOS release. Revisit only
75+
with evidence of apps the accessibility path cannot drive.
76+
- **Native by default.** The accessibility path cannot express drags, holds, or double-clicks, and
77+
apps with sparse accessibility trees still need the runner. Defaults change only with coverage
78+
evidence across app frameworks.

‎docs/adr/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
| [0028 Capability-Family Cell Vocabulary — One Runtime Source (Proposed)](0028-capability-family-cell-vocabulary.md) | adding a capability operation family, `UnavailablePlatformRuntimeFacts` / `UNAVAILABLE_CELLS`, `INTERACTOR_OPERATIONS`, and why an eight-package fan-out recurs |
3333
| [0029 Daemon Policy](0029-daemon-policy.md) | `AGENT_DEVICE_DAEMON_POLICY`, confining a daemon's commands, devices, or device shutdown, and where operator rules are enforced for batch/replay steps |
3434
| [0030 Process Lock Exclusion](0030-process-lock-exclusion.md) | process-lock publication/reclaim/release, retained mutation guards, and the single-protocol upgrade boundary |
35+
| [0031 macOS Native App Backend](0031-macos-native-app-backend.md) | `AGENT_DEVICE_MACOS_APP_BACKEND`, driving macOS app sessions without XCTest Automation Mode, and why pointer actions are accessibility actions only |
3536

3637
ADRs record *why*; the registries and gates they describe are the living source of truth — when
3738
prose and a registry disagree, the registry wins and the ADR needs a follow-up.

‎packages/contracts/src/facades/session.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,10 @@ export {
55
macOsHelperSurface,
66
macOsSurfaceBackend,
77
parseSessionSurface,
8+
readMacOsAppBackend,
89
} from '../session-surface.ts';
910
export type {
11+
MacOsAppBackend,
1012
MacOsHelperSurface,
1113
MacOsSurfaceBackend,
1214
SessionSurface,

‎packages/contracts/src/session-surface.test.ts‎

Lines changed: 48 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,59 @@ import {
33
SESSION_SURFACES,
44
macOsHelperSurface,
55
macOsSurfaceBackend,
6+
readMacOsAppBackend,
7+
type MacOsAppBackend,
68
type MacOsSurfaceBackend,
79
type SessionSurface,
810
} from './session-surface.ts';
911

10-
const EXPECTED_BACKENDS: Record<SessionSurface, MacOsSurfaceBackend> = {
11-
app: 'xctest',
12-
'frontmost-app': 'macos-helper',
13-
desktop: 'macos-helper',
14-
menubar: 'macos-helper',
12+
const EXPECTED_BACKENDS: Record<MacOsAppBackend, Record<SessionSurface, MacOsSurfaceBackend>> = {
13+
xctest: {
14+
app: 'xctest',
15+
'frontmost-app': 'macos-helper',
16+
desktop: 'macos-helper',
17+
menubar: 'macos-helper',
18+
},
19+
native: {
20+
app: 'macos-helper',
21+
'frontmost-app': 'macos-helper',
22+
desktop: 'macos-helper',
23+
menubar: 'macos-helper',
24+
},
1525
};
1626

27+
test.each(
28+
(['xctest', 'native'] as const).flatMap((appBackend) => [
29+
...SESSION_SURFACES.map(
30+
(surface) => [appBackend, surface, EXPECTED_BACKENDS[appBackend][surface]] as const,
31+
),
32+
[appBackend, undefined, EXPECTED_BACKENDS[appBackend].app] as const,
33+
]),
34+
)(
35+
'with the %s app backend the macOS %s surface is served by %s',
36+
(appBackend, surface, backend) => {
37+
expect(macOsSurfaceBackend(surface, appBackend)).toBe(backend);
38+
expect(macOsHelperSurface(surface, appBackend)).toBe(
39+
backend === 'macos-helper' ? (surface ?? 'app') : undefined,
40+
);
41+
},
42+
);
43+
1744
test.each([
18-
...SESSION_SURFACES.map((surface) => [surface, EXPECTED_BACKENDS[surface]] as const),
19-
[undefined, 'xctest'] as const,
20-
])('the macOS %s surface is served by %s', (surface, backend) => {
21-
expect(macOsSurfaceBackend(surface)).toBe(backend);
22-
expect(macOsHelperSurface(surface)).toBe(backend === 'macos-helper' ? surface : undefined);
45+
[undefined, 'xctest'],
46+
['', 'xctest'],
47+
['native', 'native'],
48+
[' Native ', 'native'],
49+
['xctest', 'xctest'],
50+
] as const)('AGENT_DEVICE_MACOS_APP_BACKEND=%j selects %s', (raw, expected) => {
51+
expect(
52+
readMacOsAppBackend((name) => {
53+
expect(name).toBe('AGENT_DEVICE_MACOS_APP_BACKEND');
54+
return raw;
55+
}),
56+
).toBe(expected);
57+
});
58+
59+
test('an unknown app backend is refused instead of falling back', () => {
60+
expect(() => readMacOsAppBackend(() => 'vision')).toThrow(/AGENT_DEVICE_MACOS_APP_BACKEND/);
2361
});

‎packages/contracts/src/session-surface.ts‎

Lines changed: 36 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -15,30 +15,54 @@ export function parseSessionSurface(value: string | undefined): SessionSurface {
1515
/** The backend that serves every operation on a macOS surface. */
1616
export type MacOsSurfaceBackend = Extract<SnapshotBackend, 'xctest' | 'macos-helper'>;
1717

18-
const MACOS_SURFACE_BACKENDS = {
19-
app: 'xctest',
18+
/**
19+
* Which backend drives a macOS app session. `xctest` is the runner under XCTest Automation Mode;
20+
* `native` drives the app through the macOS helper's accessibility actions and process-targeted
21+
* events, so the app can stay in the background and the user keeps the pointer.
22+
*/
23+
const MACOS_APP_BACKENDS = ['xctest', 'native'] as const;
24+
export type MacOsAppBackend = (typeof MACOS_APP_BACKENDS)[number];
25+
const MACOS_APP_BACKEND_ENV = 'AGENT_DEVICE_MACOS_APP_BACKEND';
26+
const MACOS_APP_BACKEND_ENUM = defineStringEnum(MACOS_APP_BACKENDS, {
27+
normalize: (raw) => raw.trim().toLowerCase(),
28+
message: (value) =>
29+
`Invalid ${MACOS_APP_BACKEND_ENV}: ${value}. Use ${MACOS_APP_BACKENDS.join('|')}.`,
30+
});
31+
32+
/** The host's app-session backend; unset selects `xctest`. */
33+
export function readMacOsAppBackend(
34+
readEnvironment: (name: string) => string | undefined,
35+
): MacOsAppBackend {
36+
const raw = readEnvironment(MACOS_APP_BACKEND_ENV);
37+
return raw === undefined || raw.trim() === '' ? 'xctest' : MACOS_APP_BACKEND_ENUM.parse(raw);
38+
}
39+
40+
const MACOS_HELPER_SURFACE_BACKENDS = {
2041
'frontmost-app': 'macos-helper',
2142
desktop: 'macos-helper',
2243
menubar: 'macos-helper',
23-
} as const satisfies Record<SessionSurface, MacOsSurfaceBackend>;
44+
} as const satisfies Record<Exclude<SessionSurface, 'app'>, MacOsSurfaceBackend>;
2445

2546
/** An absent surface is an app session, the reading every route already gives it. */
26-
export function macOsSurfaceBackend(surface: SessionSurface | undefined): MacOsSurfaceBackend {
27-
return MACOS_SURFACE_BACKENDS[surface ?? 'app'];
47+
export function macOsSurfaceBackend(
48+
surface: SessionSurface | undefined,
49+
appBackend: MacOsAppBackend,
50+
): MacOsSurfaceBackend {
51+
const resolved = surface ?? 'app';
52+
if (resolved === 'app') return appBackend === 'native' ? 'macos-helper' : 'xctest';
53+
return MACOS_HELPER_SURFACE_BACKENDS[resolved];
2854
}
2955

30-
type HelperRoutedSurface = {
31-
[S in SessionSurface]: (typeof MACOS_SURFACE_BACKENDS)[S] extends 'macos-helper' ? S : never;
32-
}[SessionSurface];
33-
3456
declare const helperSurface: unique symbol;
3557
/** A surface the owner routed to the macOS helper; only `macOsHelperSurface` produces one. */
36-
export type MacOsHelperSurface = HelperRoutedSurface & { readonly [helperSurface]: true };
58+
export type MacOsHelperSurface = SessionSurface & { readonly [helperSurface]: true };
3759

3860
export function macOsHelperSurface(
3961
surface: SessionSurface | undefined,
62+
appBackend: MacOsAppBackend,
4063
): MacOsHelperSurface | undefined {
41-
return surface !== undefined && macOsSurfaceBackend(surface) === 'macos-helper'
42-
? (surface as MacOsHelperSurface)
64+
const resolved = surface ?? 'app';
65+
return macOsSurfaceBackend(resolved, appBackend) === 'macos-helper'
66+
? (resolved as MacOsHelperSurface)
4367
: undefined;
4468
}

‎packages/platform-apple/src/interactions.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,8 @@ async function runApplePressPoint(
206206
point: { x: number; y: number },
207207
options: PressPointOptions,
208208
): Promise<Record<string, unknown>> {
209-
const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined;
209+
// The runner owner's routing: an app session reaches here only on the XCTest backend.
210+
const helper = isMacOs(device) ? macOsHelperSurface(options.surface, 'xctest') : undefined;
210211
if (helper) {
211212
return await runMacOsSurfacePress(context, point, options, helper);
212213
}

‎packages/platform-apple/src/interactor.ts‎

Lines changed: 31 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,11 @@ import {
1414
} from './runner/index.ts';
1515
import { toAppleTvRemoteButton } from '@agent-device/contracts/tv-remote';
1616
import { SCREENSHOT_FULLSCREEN_REASONS } from '@agent-device/contracts/capture';
17-
import { macOsHelperSurface, type MacOsHelperSurface } from '@agent-device/contracts/session';
17+
import {
18+
macOsHelperSurface,
19+
type MacOsHelperSurface,
20+
type SessionSurface,
21+
} from '@agent-device/contracts/session';
1822
import { DEVICE_ROTATIONS, type DeviceRotation } from '@agent-device/contracts/device';
1923
import { normalizeSnapshotScope } from '@agent-device/contracts/snapshot';
2024
import { withDiagnosticTimer } from '@agent-device/host-kit/diagnostics';
@@ -31,6 +35,8 @@ import type {
3135
SnapshotOptions,
3236
} from '@agent-device/contracts/interactor-types';
3337
import { captureMacOsSurfaceSnapshot } from './os/macos/surface-snapshot.ts';
38+
import { hostMacOsAppBackend } from './os/macos/app-backend.ts';
39+
import { macOsNativeAppInteractor } from './os/macos/native-app-interactor.ts';
3440
import {
3541
presentAppleRunnerSnapshot,
3642
readAppleSnapshotResult,
@@ -55,6 +61,9 @@ export function createAppleInteractor(
5561
);
5662
}
5763
const { overrides, runnerOpts } = iosRunnerOverrides(device, runnerContext);
64+
const appBackend = isMacOs(device) ? hostMacOsAppBackend() : 'xctest';
65+
const helperSurface = (surface: SessionSurface | undefined) =>
66+
isMacOs(device) ? macOsHelperSurface(surface, appBackend) : undefined;
5867
const interactor: Interactor = {
5968
open: (app, options) =>
6069
openIosApp(device, app, {
@@ -67,14 +76,16 @@ export function createAppleInteractor(
6776
}),
6877
openDevice: () => openIosDevice(device),
6978
close: (app) => closeIosApp(device, app, runnerOpts),
70-
screenshot: (outPath, options) => runAppleScreenshot(device, outPath, options, runnerOpts),
71-
snapshot: async (options) => await captureAppleSnapshot(device, options, runnerOpts),
79+
screenshot: (outPath, options) =>
80+
runAppleScreenshot(device, outPath, options, runnerOpts, helperSurface(options?.surface)),
81+
snapshot: async (options) =>
82+
await captureAppleSnapshot(device, options, runnerOpts, helperSurface(options?.surface)),
7283
// The live text at a point: helper for a helper-routed macOS surface, XCTest runner for
73-
// every other Apple leaf including a macOS app session.
84+
// every other Apple leaf.
7485
readTextAtPoint: async (point, options) => {
75-
const helper = isMacOs(device) ? macOsHelperSurface(options?.surface) : undefined;
86+
const helper = helperSurface(options?.surface);
7687
return helper
77-
? await readMacOsSurfaceTextAtPoint(point, helper, options?.appBundleId)
88+
? await readMacOsSurfaceTextAtPoint(point, helper, options?.appBundleId, options?.signal)
7889
: await readRunnerTextAtPoint(device, point, options, runnerOpts);
7990
},
8091
// The XCTest runner's own text reading: it observes the live accessibility hierarchy
@@ -203,16 +214,18 @@ export function createAppleInteractor(
203214
dismissAlert: (options) => actOnAppleAlert(device, runnerOpts, 'dismiss', options),
204215
...overrides,
205216
};
206-
if (!runnerProvider) return interactor;
207-
return withInjectedAppleRunnerTransport(device, runnerContext, interactor, runnerProvider);
217+
const served =
218+
appBackend === 'native' ? macOsNativeAppInteractor(interactor, runnerContext) : interactor;
219+
if (!runnerProvider) return served;
220+
return withInjectedAppleRunnerTransport(device, runnerContext, served, runnerProvider);
208221
}
209222

210223
async function captureAppleSnapshot(
211224
device: DeviceInfo,
212225
options: SnapshotOptions | undefined,
213226
runnerOpts: RunnerCallOptions,
227+
helper: MacOsHelperSurface | undefined,
214228
) {
215-
const helper = isMacOs(device) ? macOsHelperSurface(options?.surface) : undefined;
216229
if (helper) {
217230
return await captureMacOsSurfaceSnapshot({ ...options, surface: helper }, options?.signal);
218231
}
@@ -388,20 +401,24 @@ async function runAppleScreenshot(
388401
outPath: string,
389402
options: ScreenshotOptions = {},
390403
runnerOpts: RunnerCallOptions,
404+
helper: MacOsHelperSurface | undefined,
391405
): Promise<ScreenshotCaptureFacts> {
392-
const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined;
393406
if (helper) {
394407
if (options.fullscreen) {
395408
throw new AppError(
396409
'INVALID_ARGS',
397-
`screenshot --fullscreen is not accepted on the macOS ${helper} surface: it always captures the main display`,
410+
`screenshot --fullscreen is not accepted on the macOS ${helper} surface: its capture frame is fixed`,
398411
{
399412
reason: SCREENSHOT_FULLSCREEN_REASONS.macOsHelperSurfaceFixedFrame,
400413
surface: helper,
401414
},
402415
);
403416
}
404-
await runMacOsScreenshotAction(outPath, { surface: helper });
417+
await runMacOsScreenshotAction(outPath, {
418+
surface: helper,
419+
...(helper === 'app' ? { bundleId: options.appBundleId } : {}),
420+
signal: runnerOpts.signal,
421+
});
405422
return {};
406423
}
407424
if (options.captureBackend === 'runner') {
@@ -430,11 +447,13 @@ async function readMacOsSurfaceTextAtPoint(
430447
point: Point,
431448
surface: MacOsHelperSurface,
432449
appBundleId: string | undefined,
450+
signal: AbortSignal | undefined,
433451
): Promise<string | undefined> {
434452
const { runMacOsReadTextAction } = await import('./os/macos/helper.ts');
435453
const result = await runMacOsReadTextAction(point.x, point.y, {
436454
bundleId: appBundleId,
437455
surface,
456+
signal,
438457
});
439458
return result.text;
440459
}

‎packages/platform-apple/src/navigation/runtime.ts‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -183,17 +183,27 @@ export function appleNavigationFacts(device: DeviceInfo) {
183183
});
184184
}
185185

186-
/** Binds whichever navigation operations {@link appleNavigationFacts} admitted. */
186+
/**
187+
* Binds whichever navigation operations the device's admitted facts allow. The cells are those
188+
* {@link appleNavigationFacts} declares, read from the admitted facts so a refusal layered over
189+
* the leaf (a backend that cannot serve one) also withholds the binding.
190+
*/
187191
export function createAppleNavigationOperations(params: {
188192
host: Pick<PlatformRuntimeHost, 'localInteractors'>;
189193
device: DeviceInfo;
190194
signal: AbortSignal;
195+
admitted: Readonly<Record<string, RuntimeOperationFact>>;
191196
}) {
192-
const { host, device, signal } = params;
197+
const { host, device, signal, admitted } = params;
198+
const navigationKeys = Object.keys(appleNavigationFacts(device)) as Array<
199+
keyof ReturnType<typeof appleNavigationFacts>
200+
>;
193201
return bindAdmittedLocalInteractorOperations({
194202
device,
195203
signal,
196204
resolveInteractor: host.localInteractors.resolve,
197-
facts: appleNavigationFacts(device),
205+
facts: Object.fromEntries(navigationKeys.map((key) => [key, admitted[key]])) as ReturnType<
206+
typeof appleNavigationFacts
207+
>,
198208
});
199209
}

0 commit comments

Comments
 (0)