Skip to content

Commit c976a34

Browse files
janicduplessisthymikeeclaude
authored
fix(macos-helper): skip the window-sharing indicator when choosing the front window (#3965)
* fix(macos-helper): skip the window-sharing indicator when choosing the front window While a ScreenCaptureKit stream shares an app window, macOS adds a small normal-layer window with a WindowSharingSessionButton in front of it. The native backend took it for the app's front window, so scroll refused with no-scroll-bar under its center and screenshot captured the indicator. Refs #3213 * fix(macos-helper): check only the front candidate for the sharing indicator Read the accessibility tree only when the app has more than one eligible window, look only at the window that matches the front candidate, and bound the read so a hung app does not stall a capture. The screenshot help states the rule. * fix(macos-helper): bound every accessibility read of the sharing indicator probe AXUIElementSetMessagingTimeout applies only to the element it is set on, so the probe now sets it on each element it reads and stops at its own deadline. * fix(macos-helper): test the sharing indicator recognizer itself The recognizer, an AXButton child titled WindowSharingSessionButton, is now a pure function over the window's children, so a unit test covers it with an indicator and a similar ordinary window. The probe and actionWindow share one CG-to-AX frame match. The probe's doc now states why an unreadable accessibility tree keeps the frontmost window instead of refusing. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(macos-helper): read a child's title only once it is a button The recognizer now classifies one child, with the title as an autoclosure, so the probe again reads a child's title only after its role is AXButton, as it did before the recognizer became testable. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Michał Pierzchała <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 630226b commit c976a34

4 files changed

Lines changed: 105 additions & 6 deletions

File tree

‎apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift‎

Lines changed: 61 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -533,12 +533,16 @@ private func actionWindow(app: NSRunningApplication, hit: AXUIElement?) -> AXUIE
533533
}
534534
guard let front = try? frontWindowFrame(pid: app.processIdentifier) else { return nil }
535535
return windows(of: AXUIElementCreateApplication(app.processIdentifier)).first { window in
536-
guard let rect = rectAttribute(window) else { return false }
537-
return abs(rect.x - front.minX) < 1 && abs(rect.y - front.minY) < 1
538-
&& abs(rect.width - front.width) < 1 && abs(rect.height - front.height) < 1
536+
rectAttribute(window).map { isSameFrame($0, front) } ?? false
539537
}
540538
}
541539

540+
/// Whether an accessibility window frame is the frame of a WindowServer window.
541+
private func isSameFrame(_ rect: RectResponse, _ frame: CGRect) -> Bool {
542+
abs(rect.x - frame.minX) < 1 && abs(rect.y - frame.minY) < 1
543+
&& abs(rect.width - frame.width) < 1 && abs(rect.height - frame.height) < 1
544+
}
545+
542546
private func windowTitle(of element: AXUIElement) -> String? {
543547
let window = role(of: element) == "AXWindow"
544548
? element : elementAttribute(element, attribute: kAXWindowAttribute as String)
@@ -683,7 +687,21 @@ func appWindowInfo() -> [[String: Any]]? {
683687
/// none. A window list that cannot be read is a refusal, not an empty list: the caller must not
684688
/// report "no window" for a WindowServer it could not ask.
685689
func frontWindow(pid: pid_t, windowInfo: [[String: Any]]? = appWindowInfo()) throws -> OnScreenWindow? {
686-
onScreenWindows(pid: pid, windowInfo: try requireWindowInfo(windowInfo)).first
690+
let windowInfo = try requireWindowInfo(windowInfo)
691+
return frontWindow(
692+
among: onScreenWindows(pid: pid, windowInfo: windowInfo),
693+
isSharingIndicator: { isWindowSharingIndicator(pid: pid, frame: $0.bounds) }
694+
)
695+
}
696+
697+
/// The frontmost candidate that is not a window-sharing indicator. A lone window is never one: an
698+
/// indicator only appears in front of the window it belongs to.
699+
func frontWindow(
700+
among candidates: [OnScreenWindow],
701+
isSharingIndicator: (OnScreenWindow) -> Bool
702+
) -> OnScreenWindow? {
703+
guard candidates.count > 1 else { return candidates.first }
704+
return candidates.first { !isSharingIndicator($0) }
687705
}
688706

689707
func requireWindowInfo(_ windowInfo: [[String: Any]]? = appWindowInfo()) throws -> [[String: Any]] {
@@ -699,6 +717,45 @@ func requireWindowInfo(_ windowInfo: [[String: Any]]? = appWindowInfo()) throws
699717
return windowInfo
700718
}
701719

720+
/// While a ScreenCaptureKit stream shares one of an app's windows, macOS 27 adds a 66 x 20
721+
/// normal-layer window titled "Window" in front of it. Its only child is a button titled
722+
/// `WindowSharingSessionButton`. That window is a system indicator, not the app's front window.
723+
private let windowSharingIndicatorButtonTitle = "WindowSharingSessionButton"
724+
725+
/// Whether a child of a window, by its accessibility role and title, marks the window as the
726+
/// indicator. The title is read only for a button.
727+
func isWindowSharingIndicatorButton(role: String, title: @autoclosure () -> String?) -> Bool {
728+
role == "AXButton" && title() == windowSharingIndicatorButtonTitle
729+
}
730+
731+
/// `AXUIElementSetMessagingTimeout` applies only to the element it is set on, so every element the
732+
/// probe reads gets it, and the probe stops at its own deadline.
733+
private let sharingIndicatorReplyTimeoutSeconds: Float = 0.1
734+
private let sharingIndicatorProbeSeconds = 0.5
735+
736+
/// An accessibility tree that cannot be read in time, from a hung app or without Accessibility
737+
/// access, reads as no indicator, so the frontmost window stays the choice as before the indicator
738+
/// check. Refusing instead would fail screenshots, which need no accessibility read today.
739+
private func isWindowSharingIndicator(pid: pid_t, frame: CGRect) -> Bool {
740+
let deadline = Date().addingTimeInterval(sharingIndicatorProbeSeconds)
741+
func bounded(_ element: AXUIElement) -> AXUIElement? {
742+
guard Date() < deadline else { return nil }
743+
AXUIElementSetMessagingTimeout(element, sharingIndicatorReplyTimeoutSeconds)
744+
return element
745+
}
746+
guard let appElement = bounded(AXUIElementCreateApplication(pid)) else { return false }
747+
let window = windows(of: appElement).first { window in
748+
guard let window = bounded(window), let rect = rectAttribute(window) else { return false }
749+
return isSameFrame(rect, frame)
750+
}
751+
guard let window else { return false }
752+
return children(of: window).contains { child in
753+
guard let child = bounded(child) else { return false }
754+
return isWindowSharingIndicatorButton(
755+
role: role(of: child), title: stringAttribute(child, attribute: kAXTitleAttribute as String))
756+
}
757+
}
758+
702759
func frontWindowFrame(pid: pid_t, windowInfo: [[String: Any]]? = appWindowInfo()) throws -> CGRect? {
703760
try frontWindow(pid: pid, windowInfo: windowInfo)?.bounds
704761
}

‎apple/macos-helper/Tests/AgentDeviceMacOSHelperTests/AppWindowScreenshotTests.swift‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,48 @@ final class AppWindowScreenshotTests: XCTestCase {
4343
XCTAssertEqual(try frontWindowFrame(pid: 42, windowInfo: windows), front?.bounds)
4444
}
4545

46+
func testFrontWindowSkipsASharingIndicatorInFrontOfAnotherWindow() {
47+
let indicator = OnScreenWindow(
48+
number: 9, bounds: CGRect(x: 10, y: 10, width: 66, height: 20), sharingState: 1)
49+
let shared = OnScreenWindow(
50+
number: 8, bounds: CGRect(x: 0, y: 0, width: 900, height: 700), sharingState: 1)
51+
52+
XCTAssertEqual(
53+
frontWindow(among: [indicator, shared], isSharingIndicator: { $0.number == 9 })?.number, 8)
54+
XCTAssertEqual(frontWindow(among: [indicator, shared], isSharingIndicator: { _ in false })?.number, 9)
55+
}
56+
57+
func testRecognizesTheSharingIndicatorOnlyByItsSessionButton() {
58+
XCTAssertTrue(isWindowSharingIndicatorButton(role: "AXButton", title: "WindowSharingSessionButton"))
59+
60+
// A small ordinary window: title-bar buttons, or the session title on a non-button.
61+
XCTAssertFalse(isWindowSharingIndicatorButton(role: "AXButton", title: nil))
62+
XCTAssertFalse(isWindowSharingIndicatorButton(role: "AXButton", title: "Close"))
63+
XCTAssertFalse(
64+
isWindowSharingIndicatorButton(role: "AXStaticText", title: "WindowSharingSessionButton"))
65+
}
66+
67+
/// A child's title is read only once its role is a button.
68+
func testReadsNoTitleForAChildThatIsNotAButton() {
69+
var readTitle = false
70+
let isIndicator = isWindowSharingIndicatorButton(
71+
role: "AXGroup", title: { readTitle = true; return "WindowSharingSessionButton" }())
72+
73+
XCTAssertFalse(isIndicator)
74+
XCTAssertFalse(readTitle)
75+
}
76+
77+
/// The indicator check reads the app's accessibility tree; a lone window never needs it.
78+
func testFrontWindowAsksNothingAboutALoneWindow() {
79+
let lone = OnScreenWindow(
80+
number: 8, bounds: CGRect(x: 0, y: 0, width: 900, height: 700), sharingState: 1)
81+
var asked = false
82+
let front = frontWindow(among: [lone], isSharingIndicator: { _ in asked = true; return true })
83+
84+
XCTAssertEqual(front?.number, 8)
85+
XCTAssertFalse(asked)
86+
}
87+
4688
func testRefusesWhenOnlyInvisibleOrEmptyWindowsRemain() throws {
4789
let windows = [
4890
windowInfo(id: 1, onScreen: false),

‎packages/command-surface/src/capture/screenshot.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ export const screenshotCommandFacet = defineCommandFacet({
8484
text: {
8585
summary: 'Capture a screenshot',
8686
cliDetail:
87-
"Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. Use --screen-mask black to fill the iOS simulator screen outline, including rounded corners, with black for this capture. It requires simctl: physical devices, tvOS and visionOS simulators, other platforms, provider captures and XCTest runner captures are refused. A masked capture fails if simctl fails instead of falling back to an unmasked runner capture. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --crop-on <selector> to crop the capture to the frame the selector resolves on the same screen (currently iOS simulators, Android emulators, and CoreDevice physical iPhones), --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator or Android chrome. Physical iPhone crop verification covers portrait CoreDevice/devicectl captures only. XCTest, runner fallback, and physical iPad crops are refused; landscape iPhone crop is unverified. Android captures immediately by default; --normalize-status-bar also waits for transient UI to fade, and --no-stabilize keeps capture immediate. With the native macOS backend, the front window is the app's frontmost visible, nonempty normal-level (layer-0) window on the current Space: floating palettes and app-modal alerts at window level 8 are skipped, and a sheet is captured as its own window. On macOS, any --surface other than app (desktop, menubar, frontmost-app) always captures the main display and refuses an explicit --fullscreen. On single-display iOS simulators and Android devices, the result reports displayRotation, the rotation the display rendered the capture in, when the device returns a reading in time; otherwise the field is absent. On iOS simulators a screenshot waits for that simulator's status bar to be free, and a request its caller cancelled while it waits reports the cancellation instead of capturing.",
87+
"Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. Use --screen-mask black to fill the iOS simulator screen outline, including rounded corners, with black for this capture. It requires simctl: physical devices, tvOS and visionOS simulators, other platforms, provider captures and XCTest runner captures are refused. A masked capture fails if simctl fails instead of falling back to an unmasked runner capture. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --crop-on <selector> to crop the capture to the frame the selector resolves on the same screen (currently iOS simulators, Android emulators, and CoreDevice physical iPhones), --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator or Android chrome. Physical iPhone crop verification covers portrait CoreDevice/devicectl captures only. XCTest, runner fallback, and physical iPad crops are refused; landscape iPhone crop is unverified. Android captures immediately by default; --normalize-status-bar also waits for transient UI to fade, and --no-stabilize keeps capture immediate. With the native macOS backend, the front window is the app's frontmost visible, nonempty normal-level (layer-0) window on the current Space: floating palettes and app-modal alerts at window level 8 are skipped, as is the small indicator window macOS adds in front of a window that a screen-sharing stream captures, and a sheet is captured as its own window. On macOS, any --surface other than app (desktop, menubar, frontmost-app) always captures the main display and refuses an explicit --fullscreen. On single-display iOS simulators and Android devices, the result reports displayRotation, the rotation the display rendered the capture in, when the device returns a reading in time; otherwise the field is absent. On iOS simulators a screenshot waits for that simulator's status bar to be free, and a request its caller cancelled while it waits reports the cancellation instead of capturing.",
8888
},
8989
metadata: screenshotCommandMetadata,
9090
run: (client, input) => client.capture.screenshot(input),

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -363,7 +363,7 @@ With the native backend, each app you open on the `app` surface is a device of i
363363
364364
Copies of one app need distinct bundle ids. Screenshots run one at a time across the Mac. Use one agent-device version per claims directory.
365365
366-
- `click`, `press`, and `fill` use accessibility actions on the element at the target point, which must be inside one of the app's on-screen windows, not its menu bar. `click --button secondary` opens the context menu of the nearest element at the point that offers one (`AXShowMenu`); the open menu shows in the next `snapshot`, and `click` on one of its items picks it. It fails with `UNSUPPORTED_OPERATION` (`helperReason: no-context-menu`) when no element there offers a menu (`no-accessible-target` when no element is at the point). It fails with `COMMAND_FAILED` and `reason: context-menu-not-observed` when the app's window list was readable but showed no new menu window, and with `reason: window-list-unreadable` when the window list could not be read before the action or by any read during the wait for the menu. `type` inserts at the focused control or falls back to key events. `scroll` moves the scroll bar under the front window's center. Electron and other Chromium-based desktop apps expose their web content in the snapshot: the first `snapshot` turns the app's Chromium accessibility tree on, and a snapshot warns when that tree did not populate. `screenshot` captures only the app's front window, even when it is covered. The front window is the app's frontmost visible, nonempty normal-level (layer-0) window on the current Space: floating palettes and app-modal alerts at window level 8 are skipped, and a sheet is captured as its own window. If that window cannot be captured, `screenshot` fails; it never captures another window.
366+
- `click`, `press`, and `fill` use accessibility actions on the element at the target point, which must be inside one of the app's on-screen windows, not its menu bar. `click --button secondary` opens the context menu of the nearest element at the point that offers one (`AXShowMenu`); the open menu shows in the next `snapshot`, and `click` on one of its items picks it. It fails with `UNSUPPORTED_OPERATION` (`helperReason: no-context-menu`) when no element there offers a menu (`no-accessible-target` when no element is at the point). It fails with `COMMAND_FAILED` and `reason: context-menu-not-observed` when the app's window list was readable but showed no new menu window, and with `reason: window-list-unreadable` when the window list could not be read before the action or by any read during the wait for the menu. `type` inserts at the focused control or falls back to key events. `scroll` moves the scroll bar under the front window's center. Electron and other Chromium-based desktop apps expose their web content in the snapshot: the first `snapshot` turns the app's Chromium accessibility tree on, and a snapshot warns when that tree did not populate. `screenshot` captures only the app's front window, even when it is covered. The front window is the app's frontmost visible, nonempty normal-level (layer-0) window on the current Space: floating palettes and app-modal alerts at window level 8 are skipped, as is the small indicator window macOS adds in front of a window that a screen-sharing stream captures, and a sheet is captured as its own window. If that window cannot be captured, `screenshot` fails; it never captures another window.
367367
- `record`, `prepare`, `back`, `longpress`, `--hold-ms`, `--double-tap`, gestures, and clicks on an element with no accessibility action fail with `UNSUPPORTED_OPERATION` (`reason: unsupported-device-backend`). Use the default XCTest backend for those.
368368
369369
```bash

0 commit comments

Comments
 (0)