You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dd12a41
Browse filesBrowse the repository at this point in the historyBrowse files
fix(apple-runner): refuse a macOS read of a stopped app instead of launching it
`notRunningRefusal` was `#if os(iOS)`, so on macOS a `.existingApp` read of a
session app that was not running fell through to `activateTarget` and bare-launched
it — which on macOS also takes the user's frontmost app away. The comment claiming
only iOS can read an app's state without launching it does not hold: measured on a
macOS host, a fresh `XCUIApplication(bundleIdentifier:)` for a stopped app answers
`.notRunning` and no process appears.
The macOS host lane now proves the refusal is reachable there (a user-level read, a
mutation's leading read, and the selector-resolving read) and that a command whose
job is to bring the app up still does.
Measures from the same host lane also settle two questions the runner's activation
sites raise, and are recorded where a reader will find them. An app-targeted
`screenshot` on this backend captures the window's screen region, so its
`activate()` is load-bearing for what the pixels show: backgrounded, it returned the
occluding app's windows inside the session window's own rect. And a backgrounded
macOS app answers a divergent tree — 146 nodes against 143, with 38 of the 143
name-matched nodes reporting a different rect, one button 36.5x16 against 65.5x24,
y drifting up to 60 px, and the scroll indicator's extent 260.5 against 273 — so the
foreground repair stays load-bearing for reads, because the rect is what the
daemon's occlusion and offscreen guards decide on.
Documents `AGENT_DEVICE_MACOS_APP_BACKEND` in versioned `help macos` and names in
the commands docs which macOS commands still activate the app or move the real
pointer.
Partial #3254
Copy file name to clipboardExpand all lines: src/commands/schema/cli-help.ts
+7-1Lines changed: 7 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -739,7 +739,13 @@ Rules:
739
739
Context menus are not ambient UI: secondary-click a visible target, then re-snapshot and use the new menu-item refs.
740
740
Do not let iOS simulator-set scoping hide macOS desktop targets.
741
741
Prefer refs/selectors over raw coordinates.
742
-
macOS snapshot rects are window-space; use current refs or overlay refs instead of guessing coordinates.`,
742
+
macOS snapshot rects are window-space; use current refs or overlay refs instead of guessing coordinates.
743
+
744
+
Activation and the pointer (what a run does to the user's screen):
745
+
An app session on the default XCTest backend brings the session app forward before a command that names it, and shows the Automation Mode overlay. A read of a session app that is not running answers APP_NOT_RUNNING rather than starting it.
746
+
AGENT_DEVICE_MACOS_APP_BACKEND=native (a daemon setting; restart the daemon after changing it) serves app sessions through the macOS helper's accessibility actions instead: no runner, no overlay, and snapshot, screenshot, get, click, press, fill, type, and scroll leave the app where it sits. A command only the runner can serve refuses with UNSUPPORTED_OPERATION (details.reason unsupported-device-backend).
747
+
press and click on the frontmost-app and menubar surfaces post synthetic mouse events at screen coordinates on either backend: the real pointer moves, and macOS activates the app that receives them.
748
+
On the XCTest backend an app-targeted screenshot activates the app first, because XCTest captures the screen region the window occupies and a background app would come back with whatever is drawn over it.`,
743
749
},
744
750
web: {
745
751
summary: 'Minimal browser workflow with the managed web backend',
- Use `click --button secondary`for context menus on macOS, then run `snapshot -i` again.
349
349
- On `frontmost-app` and `menubar` surfaces, `press` and `click` post synthetic mouse events through the macOS helper (the `desktop` surface inspects only): `--hold-ms` is how long the button stays down (at least 40 ms, 60 ms by default, because AppKit drops a release posted in the same tick as its press), `--count` is that many independent clicks (apps that detect a double-click by timing, such as Finder, still read two clicks at the default interval as one; pass an `--interval-ms` longer than the system double-click time to keep them apart), and `--double-tap` posts each click as a double-click pair. A long schedule such as `--hold-ms 10000 --count 4` is given the time it needs, and a helper stopped mid-hold — by a cancelled request, a dropped client, or its deadline — releases the button before it exits. `--jitter-px` is not applied on these surfaces.
350
350
- With `AGENT_DEVICE_MACOS_APP_BACKEND=native`, macOS app sessions run without XCTest: no Automation Mode overlay, the app can stay behind other windows, and the real pointer stays with you while a drawn pointer shows each action. `click`, `press`, and `fill` use accessibility actions on the element at their target point, which must lie inside one of the app's own on-screen windows and not on its menu bar; `type` inserts at the focused control and falls back to key events sent to the app; `scroll` moves the accessibility scroll bar under the front window's center. Chromium-based apps expose their full tree. The daemon then never starts the Apple runner on macOS: `record`, `prepare`, `back`, `longpress` and `--hold-ms`, `--double-tap`, `--button secondary`, and gestures are refused with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`), as is a click on an element with no accessibility action. Use the default XCTest backend for those. `screenshot` captures only the app's front window, even when other windows cover it.
351
+
- Which macOS commands still take the app forward, or move the real pointer:
352
+
- Default (XCTest) app sessions: a command that resolves or drives the session app activates it first unless it is already frontmost, the runner moves the real pointer for taps and gestures, and the runner shows the Automation Mode overlay while the session lives. A read (`snapshot`, `wait`, `is`, `get`, a reading `find`, and an interaction's leading read) of a session app that is not running answers `APP_NOT_RUNNING` rather than starting it. An app-targeted `screenshot` also activates the app, because XCTest captures the screen region the window occupies — a background window would come back with whatever is drawn over it.
353
+
- `AGENT_DEVICE_MACOS_APP_BACKEND=native` app sessions: `snapshot`, `screenshot`, `get`, `click`, `press`, `fill`, `type`, and `scroll` act through accessibility on the app's own windows, so none of them activates the app or moves the real pointer. `open` still launches or reopens the app, which brings it forward.
354
+
- `press` and `click` on `frontmost-app` and `menubar` surfaces, on either backend: these post synthetic mouse events at screen coordinates through the macOS helper (`CGEvent.post(tap: .cghidEventTap)`in`MouseClickDelivery.swift`). The real pointer moves to the point, the click lands on whichever window is topmost there, and macOS activates the app that receives it.
351
355
- Mobile-only helpers are unsupported on macOS: `boot`, `shutdown`, `home`, `orientation`, `app-switcher`, `action-button`, `fold`, `install`, `reinstall`, `install-from-source`, and `push`.
0 commit comments