You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit edf873e
Browse filesBrowse the repository at this point in the historyBrowse files
fix(macos-helper): act only inside the session app's windows on the native backend
The native backend's hit test runs on the app's accessibility tree, which also
holds the app's menu bar, and the menu bar's Apple menu acts for the whole Mac
(Sleep, Lock Screen, Recent Items). press, focus and fill now refuse a point
outside every window the app owns, and a resolved target inside its menu bar,
with no-accessible-target.
Copy file name to clipboardExpand all lines: docs/adr/0031-macos-native-app-backend.md
+4-2Lines changed: 4 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -52,8 +52,10 @@ were accepted by Calculator and by Electron apps in the background. A fallback t
52
52
observed to work would turn "nothing happened" into success, so pointer actions have no event
53
53
fallback.
54
54
55
-
**Target resolution.** The helper hit-tests inside the session app (other apps' windows above it do
56
-
not answer) and walks at most four ancestors for a text input or a pressable control role.
55
+
**Target resolution.** The helper acts only on a point inside one of the session app's own on-screen
56
+
windows (its menus and sheets included) and never on its menu bar, whose Apple menu acts for the whole Mac
57
+
(Sleep, Lock Screen, Recent Items); other points are refused with `no-accessible-target`. It
58
+
hit-tests inside the session app (other apps' windows above it do not answer) and walks at most four ancestors for a text input or a pressable control role.
57
59
Chromium answers a hit test with wrapper groups that all claim `AXPress`, so when the chain names
58
60
no control the helper picks the smallest such element whose frame contains the point, searching
59
61
only the window the hit landed in (the app's front on-screen window when the hit names none). A
- Prefer selector or `@ref`-driven interactions on macOS. Window position can shift between runs, so raw x/y point commands are less stable than snapshot-derived targets.
346
346
- Use `click --button secondary`for context menus on macOS, then run `snapshot -i` again.
347
347
- On `frontmost-app` and `menubar` surfaces, `press` and `click` post synthetic mouse events through the macOS helper (the `desktop` surface inspects only): `--hold-ms` is how long the button stays down (at least 40 ms, 60 ms by default, because AppKit drops a release posted in the same tick as its press), `--count` is that many independent clicks (apps that detect a double-click by timing, such as Finder, still read two clicks at the default interval as one; pass an `--interval-ms` longer than the system double-click time to keep them apart), and `--double-tap` posts each click as a double-click pair. A long schedule such as `--hold-ms 10000 --count 4` is given the time it needs, and a helper stopped mid-hold — by a cancelled request, a dropped client, or its deadline — releases the button before it exits. `--jitter-px` is not applied on these surfaces.
348
-
- With `AGENT_DEVICE_MACOS_APP_BACKEND=native`, macOS app sessions run without XCTest: no Automation Mode overlay, the app can stay behind other windows, and the real pointer stays with you while a drawn pointer shows each action. `click`, `press`, and `fill` use accessibility actions on the element at their target point;`type` inserts at the focused control and falls back to key events sent to the app;`scroll` moves the accessibility scroll bar under the front window's center. Chromium-based apps expose their full tree. The daemon then never starts the XCTest runner on macOS: `record`, `prepare`, `back`, `longpress` and `--hold-ms`, `--double-tap`, `--button secondary`, and gestures are refused with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`), as is a click on an element with no accessibility action. Use the default XCTest backend for those. `screenshot` captures only the app's front window, even when other windows cover it.
348
+
- With `AGENT_DEVICE_MACOS_APP_BACKEND=native`, macOS app sessions run without XCTest: no Automation Mode overlay, the app can stay behind other windows, and the real pointer stays with you while a drawn pointer shows each action. `click`, `press`, and `fill` use accessibility actions on the element at their target point, which must lie inside one of the app's own on-screen windows and not on its menu bar; `type` inserts at the focused control and falls back to key events sent to the app; `scroll` moves the accessibility scroll bar under the front window's center. Chromium-based apps expose their full tree. The daemon then never starts the XCTest runner on macOS: `record`, `prepare`, `back`, `longpress` and `--hold-ms`, `--double-tap`, `--button secondary`, and gestures are refused with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`), as is a click on an element with no accessibility action. Use the default XCTest backend for those. `screenshot` captures only the app's front window, even when other windows cover it.
349
349
- Mobile-only helpers remain unsupported on macOS: `boot`, `shutdown`, `home`, `orientation`, `app-switcher`, `action-button`, `fold`, `install`, `reinstall`, `install-from-source`, and `push`.
0 commit comments