Skip to content

Commit edf873e

Browse files
fix(macos-helper): act only inside the session app's windows on the native backend
The native backend's hit test runs on the app's accessibility tree, which also holds the app's menu bar, and the menu bar's Apple menu acts for the whole Mac (Sleep, Lock Screen, Recent Items). press, focus and fill now refuse a point outside every window the app owns, and a resolved target inside its menu bar, with no-accessible-target.
1 parent 474b464 commit edf873e

3 files changed

Lines changed: 53 additions & 6 deletions

File tree

‎apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift‎

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -95,10 +95,13 @@ func pressInBackground(
9595
)
9696
}
9797
let point = CGPoint(x: request.x, y: request.y)
98+
try requireAppWindowPoint(point, app: app)
9899
cursor?.move(to: point)
99-
guard let target = resolvePressTarget(app: app, point: point),
100-
let mechanism = perform(target)
101-
else {
100+
let target = resolvePressTarget(app: app, point: point)
101+
if let target, isInMenuBar(target.element) {
102+
throw refusal(.noAccessibleTarget, "the point is on the app's menu bar", app: app)
103+
}
104+
guard let target, let mechanism = perform(target) else {
102105
throw refusal(.noAccessibleTarget, "no pressable accessibility element at the point", app: app)
103106
}
104107
var clicks = 1
@@ -165,12 +168,16 @@ func fillInBackground(
165168
app: NSRunningApplication,
166169
cursor: GhostCursor?
167170
) throws -> BackgroundTextResponse {
171+
try requireAppWindowPoint(point, app: app)
168172
cursor?.move(to: point)
169173
let hit = elementAtPoint(in: app, point: point)
170174
let chain = hit.map(pressSearchChain) ?? []
171175
let fallback = actionWindow(app: app, hit: hit).flatMap { window in
172176
smallestElement(in: window, containing: point, where: isTextInput)
173177
}
178+
if let input = chain.first(where: isTextInput) ?? fallback, isInMenuBar(input) {
179+
throw refusal(.noAccessibleTarget, "the point is on the app's menu bar", app: app)
180+
}
174181
guard let input = chain.first(where: isTextInput) ?? fallback,
175182
isAttributeSettable(input, attribute: kAXValueAttribute as String)
176183
else {
@@ -294,6 +301,44 @@ private func perform(_ target: PressTarget) -> BackgroundDeliveryMechanism? {
294301
}
295302
}
296303

304+
/// The app's accessibility tree also holds its menu bar, whose Apple menu acts for the whole Mac
305+
/// (Sleep, Lock Screen, Recent Items). A point action therefore lands only inside one of the app's
306+
/// own on-screen windows, its menus and sheets included, and never on a menu bar element: the app
307+
/// owns its menu bar's window too, so the window check alone does not exclude it.
308+
private func requireAppWindowPoint(_ point: CGPoint, app: NSRunningApplication) throws {
309+
guard
310+
let info = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID)
311+
as? [[String: Any]]
312+
else {
313+
throw refusal(.noAccessibleTarget, "the app's on-screen windows could not be read", app: app)
314+
}
315+
let inAppWindow = info.contains { entry in
316+
guard (entry[kCGWindowOwnerPID as String] as? Int32) == app.processIdentifier,
317+
let boundsDict = entry[kCGWindowBounds as String] as? NSDictionary,
318+
let bounds = CGRect(dictionaryRepresentation: boundsDict as CFDictionary)
319+
else {
320+
return false
321+
}
322+
return bounds.contains(point)
323+
}
324+
guard inAppWindow else {
325+
throw refusal(.noAccessibleTarget, "the point is outside the app's on-screen windows", app: app)
326+
}
327+
}
328+
329+
private func isInMenuBar(_ element: AXUIElement) -> Bool {
330+
var current: AXUIElement? = element
331+
var depth = 0
332+
while let node = current, depth < 32 {
333+
let nodeRole = role(of: node)
334+
if nodeRole == "AXMenuBar" || nodeRole == "AXMenuBarItem" { return true }
335+
if nodeRole == "AXApplication" { return false }
336+
current = elementAttribute(node, attribute: kAXParentAttribute as String)
337+
depth += 1
338+
}
339+
return false
340+
}
341+
297342
/// The app's own hit test, scoped to the app so windows of other apps above it do not answer.
298343
private func elementAtPoint(in app: NSRunningApplication, point: CGPoint) -> AXUIElement? {
299344
let appElement = AXUIElementCreateApplication(app.processIdentifier)

‎docs/adr/0031-macos-native-app-backend.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,10 @@ were accepted by Calculator and by Electron apps in the background. A fallback t
5252
observed to work would turn "nothing happened" into success, so pointer actions have no event
5353
fallback.
5454

55-
**Target resolution.** The helper hit-tests inside the session app (other apps' windows above it do
56-
not answer) and walks at most four ancestors for a text input or a pressable control role.
55+
**Target resolution.** The helper acts only on a point inside one of the session app's own on-screen
56+
windows (its menus and sheets included) and never on its menu bar, whose Apple menu acts for the whole Mac
57+
(Sleep, Lock Screen, Recent Items); other points are refused with `no-accessible-target`. It
58+
hit-tests inside the session app (other apps' windows above it do not answer) and walks at most four ancestors for a text input or a pressable control role.
5759
Chromium answers a hit test with wrapper groups that all claim `AXPress`, so when the chain names
5860
no control the helper picks the smallest such element whose frame contains the point, searching
5961
only the window the hit landed in (the app's front on-screen window when the hit names none). A

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -345,7 +345,7 @@ agent-device snapshot -i --platform apple --target desktop
345345
- Prefer selector or `@ref`-driven interactions on macOS. Window position can shift between runs, so raw x/y point commands are less stable than snapshot-derived targets.
346346
- Use `click --button secondary` for context menus on macOS, then run `snapshot -i` again.
347347
- On `frontmost-app` and `menubar` surfaces, `press` and `click` post synthetic mouse events through the macOS helper (the `desktop` surface inspects only): `--hold-ms` is how long the button stays down (at least 40 ms, 60 ms by default, because AppKit drops a release posted in the same tick as its press), `--count` is that many independent clicks (apps that detect a double-click by timing, such as Finder, still read two clicks at the default interval as one; pass an `--interval-ms` longer than the system double-click time to keep them apart), and `--double-tap` posts each click as a double-click pair. A long schedule such as `--hold-ms 10000 --count 4` is given the time it needs, and a helper stopped mid-hold — by a cancelled request, a dropped client, or its deadline — releases the button before it exits. `--jitter-px` is not applied on these surfaces.
348-
- With `AGENT_DEVICE_MACOS_APP_BACKEND=native`, macOS app sessions run without XCTest: no Automation Mode overlay, the app can stay behind other windows, and the real pointer stays with you while a drawn pointer shows each action. `click`, `press`, and `fill` use accessibility actions on the element at their target point; `type` inserts at the focused control and falls back to key events sent to the app; `scroll` moves the accessibility scroll bar under the front window's center. Chromium-based apps expose their full tree. The daemon then never starts the XCTest runner on macOS: `record`, `prepare`, `back`, `longpress` and `--hold-ms`, `--double-tap`, `--button secondary`, and gestures are refused with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`), as is a click on an element with no accessibility action. Use the default XCTest backend for those. `screenshot` captures only the app's front window, even when other windows cover it.
348+
- With `AGENT_DEVICE_MACOS_APP_BACKEND=native`, macOS app sessions run without XCTest: no Automation Mode overlay, the app can stay behind other windows, and the real pointer stays with you while a drawn pointer shows each action. `click`, `press`, and `fill` use accessibility actions on the element at their target point, which must lie inside one of the app's own on-screen windows and not on its menu bar; `type` inserts at the focused control and falls back to key events sent to the app; `scroll` moves the accessibility scroll bar under the front window's center. Chromium-based apps expose their full tree. The daemon then never starts the XCTest runner on macOS: `record`, `prepare`, `back`, `longpress` and `--hold-ms`, `--double-tap`, `--button secondary`, and gestures are refused with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`), as is a click on an element with no accessibility action. Use the default XCTest backend for those. `screenshot` captures only the app's front window, even when other windows cover it.
349349
- Mobile-only helpers remain unsupported on macOS: `boot`, `shutdown`, `home`, `orientation`, `app-switcher`, `action-button`, `fold`, `install`, `reinstall`, `install-from-source`, and `push`.
350350
351351
Recommended loops:

0 commit comments

Comments
 (0)