Skip to content

Commit 78212ad

Browse files
f-trycuaclaude
andauthored
feat(cua-driver): capture GNOME windows from their own actor (WinRects v10) (trycua#4882)
* feat(cua-driver): capture GNOME windows from their own actor (WinRects v10) On GNOME Wayland, window screenshots were crops of the Shell stage. GetRects proves a window's geometry, not that nothing covers it, so a covered window's screenshot showed the app on top. The WinRects helper gains two additive methods: - CaptureWindow(id, pid), API 9: snapshot the window's own actor with paint_to_content. Covered windows keep their content, nothing is activated, and a 1x1 transparent clone keeps Mutter painting a fully covered window for 4 s after the last capture. Refused while locked, for minimized windows, and above 32 MiB. - CaptureWindowPreview(id, pid, max), API 10: the same capture, downscaled on the GPU, returning the full-size dimensions too. The driver uses CaptureWindow whenever the attested helper reports API 9 or newer, and then never falls back to a stage crop. Older helpers keep the stage crop, so users who have not reinstalled the helper keep window screenshots. A screenshot-only get_window_state with a size cap uses CaptureWindowPreview (falling back to the full capture on older helpers), and GNOME's per-capture Creation Time PNG chunk is removed so unchanged preview frames compare equal. list_windows asks a verified GNOME helper first instead of probing other compositor protocols. Versioning: the UUID stays winrects@cua. One UUID means one installed copy per user; two UUIDs would race for org.cua.WinRects. API numbers 9 and 10 match T3 Code's bundled helper exactly, so either copy serves both drivers. install.sh and both cua-driver installers no longer downgrade a newer installed helper (install.sh --force still does). The helper README documents the rules and the version table. Ported from T3 Code's Linux patch to Cua Driver 0.34.0 (pingdotgg/t3code#16975, MIT, Copyright (c) 2026 T3 Tools Inc.). The stage-crop fallback for older helpers and the no-downgrade installers are new here. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(cua-driver): credit every T3 Code Linux port in THIRD_PARTY_NOTICES Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 48cc491 commit 78212ad

13 files changed

Lines changed: 548 additions & 36 deletions

File tree

‎libs/cua-driver/rust/THIRD_PARTY_NOTICES.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,14 @@ In `cursor-overlay`, the motion knobs, their defaults, the arc formula and the
1111
Dubins glide originate in that Swift code; trope-cua is credited below only for
1212
the items listed.
1313

14-
| Project | Copyright | Used in | What |
15-
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
16-
| [trope-cua](https://github.com/voctory/trope-cua) | Copyright (c) 2026 Victor Vannara | `crates/platform-windows` (`src/overlay.rs`, `src/input/`, `src/tools/impl_.rs`); `crates/cursor-overlay/src/motion.rs`, `crates/cursor-overlay/src/bezier.rs` (specific items only) | Windows cursor overlay, background input and layered pixel-click dispatch; in `cursor-overlay`, the `press_duration_ms` knob, the 80 ms dwell default, the former 20 s idle default, the `with_overrides` method shape and the cached 32-point arc-length helper |
17-
| [Interface-Agent](https://github.com/francedot/Interface-Agent) | Copyright (c) 2024 Francesco Bonacci | `crates/platform-windows` (`src/uia/`, `src/win32/`, `src/tools/impl_.rs`) | Windows UI Automation tree walk, app and window enumeration, InvokePattern click and ValuePattern set-value |
18-
| [yabai](https://github.com/koekeishiya/yabai) | Copyright (c) 2019 Åsmund Vikane | `crates/platform-macos/src/input/skylight.rs` (`activate_without_raise`) | The focus-without-raise SkyLight event-record sequence |
19-
| [QuickJS-ng](https://github.com/quickjs-ng/quickjs) | Copyright (c) 2017-2026 Fabrice Bellard; (c) 2017-2024 Charlie Gordon; (c) 2023-2026 Ben Noordhuis; (c) 2023-2026 Saúl Ibarra Corretgé | `crates/cua-driver-core/src/script_tool/` (compiled in through rquickjs-sys) | The JavaScript engine behind the experimental `run_script` tool |
20-
| [rquickjs](https://github.com/DelSkayn/rquickjs) | Copyright (c) 2020 Mees Delzenne; (c) 2025 Rquickjs Contributors | `crates/cua-driver-core/src/script_tool/` | Rust bindings to QuickJS-ng |
21-
| [T3 Code](https://github.com/pingdotgg/t3code) | Copyright (c) 2026 T3 Tools Inc. | `crates/cua-driver-sdk/src/embedded.rs` (`allowed_environment_name` and its test) | Linux fixes ported from T3 Code's patch to Cua Driver 0.34.0 ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975), `patches/cua-driver-linux/0.34.0.patch`): the `CUA_DRIVER_RS_ENABLE_WAYLAND` embedded-host pass-through |
14+
| Project | Copyright | Used in | What |
15+
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
16+
| [trope-cua](https://github.com/voctory/trope-cua) | Copyright (c) 2026 Victor Vannara | `crates/platform-windows` (`src/overlay.rs`, `src/input/`, `src/tools/impl_.rs`); `crates/cursor-overlay/src/motion.rs`, `crates/cursor-overlay/src/bezier.rs` (specific items only) | Windows cursor overlay, background input and layered pixel-click dispatch; in `cursor-overlay`, the `press_duration_ms` knob, the 80 ms dwell default, the former 20 s idle default, the `with_overrides` method shape and the cached 32-point arc-length helper |
17+
| [Interface-Agent](https://github.com/francedot/Interface-Agent) | Copyright (c) 2024 Francesco Bonacci | `crates/platform-windows` (`src/uia/`, `src/win32/`, `src/tools/impl_.rs`) | Windows UI Automation tree walk, app and window enumeration, InvokePattern click and ValuePattern set-value |
18+
| [yabai](https://github.com/koekeishiya/yabai) | Copyright (c) 2019 Åsmund Vikane | `crates/platform-macos/src/input/skylight.rs` (`activate_without_raise`) | The focus-without-raise SkyLight event-record sequence |
19+
| [QuickJS-ng](https://github.com/quickjs-ng/quickjs) | Copyright (c) 2017-2026 Fabrice Bellard; (c) 2017-2024 Charlie Gordon; (c) 2023-2026 Ben Noordhuis; (c) 2023-2026 Saúl Ibarra Corretgé | `crates/cua-driver-core/src/script_tool/` (compiled in through rquickjs-sys) | The JavaScript engine behind the experimental `run_script` tool |
20+
| [rquickjs](https://github.com/DelSkayn/rquickjs) | Copyright (c) 2020 Mees Delzenne; (c) 2025 Rquickjs Contributors | `crates/cua-driver-core/src/script_tool/` | Rust bindings to QuickJS-ng |
21+
| [T3 Code](https://github.com/pingdotgg/t3code) | Copyright (c) 2026 T3 Tools Inc. | `crates/cua-driver-sdk/src/embedded.rs` (`allowed_environment_name`); `crates/platform-linux/src/wayland/shell_helper.rs` and `shell_helper/transport.rs`; `crates/platform-linux/src/wayland/libei.rs` (input acknowledgement); `crates/platform-linux/src/wayland/mod.rs` and `src/tools/impl_.rs` (GNOME window capture); `../wayland-helper/winrects@cua/` (`CaptureWindow`, `CaptureWindowPreview`) | Linux fixes ported from T3 Code's patch to Cua Driver 0.34.0 ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975), `patches/cua-driver-linux/0.34.0.patch`): the `CUA_DRIVER_RS_ENABLE_WAYLAND` embedded-host pass-through, the persistent session-bus transport for the GNOME helper, waiting for EIS to acknowledge libei input, and per-window GNOME capture with compositor-side preview downscaling (helper API 9 and 10) |
2222

2323
## MIT License
2424

‎libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1371,20 +1371,34 @@ impl Tool for GetWindowStateTool {
13711371
window_overlays(pid, xid)
13721372
};
13731373
let screenshot = if should_capture {
1374-
let captured = if overlays.covers_window {
1374+
// A screenshot-only read with a size cap (a live preview) asks
1375+
// a GNOME helper with API 10 to downscale in the compositor,
1376+
// and keeps the window's full size for frame_scale.
1377+
let preview = if !want_tree && max_dim > 0 && crate::wayland::is_wayland() {
1378+
crate::wayland::shell_helper::screenshot_window_preview(xid, pid, max_dim)
1379+
} else {
1380+
None
1381+
};
1382+
let captured = if let Some((png, width, height)) = preview {
1383+
Ok((png, Some((width, height))))
1384+
} else if overlays.covers_window {
13751385
overlays
13761386
.window_rect
13771387
.ok_or_else(|| anyhow::anyhow!("window geometry unavailable"))
13781388
.and_then(|(x, y, w, h)| {
13791389
crate::capture::screenshot_root_region_png(x, y, w, h)
13801390
})
13811391
.or_else(|_| crate::wayland::screenshot_dispatch_with_pid(xid, pid))
1392+
.map(|png| (png, None))
13821393
} else {
1383-
crate::wayland::screenshot_dispatch_with_pid(xid, pid)
1394+
crate::wayland::screenshot_dispatch_with_pid(xid, pid).map(|png| (png, None))
13841395
};
13851396
match captured {
1386-
Ok(raw) => {
1387-
let (orig_w, orig_h) = crate::capture::png_dimensions_pub(&raw)?;
1397+
Ok((raw, original_size)) => {
1398+
let (orig_w, orig_h) = match original_size {
1399+
Some(size) => size,
1400+
None => crate::capture::png_dimensions_pub(&raw)?,
1401+
};
13881402
let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;
13891403
let (w, h) = crate::capture::png_dimensions_pub(&png)?;
13901404
// An explicit per-call max_image_dimension (0 = native)

‎libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1044,6 +1044,10 @@ fn wayland_window_crop(xid: u64) -> anyhow::Result<WaylandWindowCrop> {
10441044
window.visible,
10451045
);
10461046
}
1047+
// GNOME helpers before API 9 have no per-window capture, so the stage is
1048+
// cropped to the window's frame. GetRects proves geometry, not occlusion:
1049+
// a covering window shows in the crop. API 9+ helpers never reach this
1050+
// (see `screenshot_dispatch_for_pid`).
10471051
if let Some(window) = shell_helper::list_windows(None)
10481052
.and_then(|windows| windows.into_iter().find(|window| window.xid == xid))
10491053
{
@@ -1106,6 +1110,23 @@ fn screenshot_dispatch_for_pid(xid: u64, pid: Option<u32>) -> anyhow::Result<Vec
11061110
)
11071111
});
11081112
}
1113+
if is_wayland() {
1114+
// A GNOME helper with API 9+ snapshots the window's own actor, so a
1115+
// covering window never shows. It is then the only GNOME route: a
1116+
// refused capture (minimized window, locked screen) must not fall
1117+
// back to a stage crop that could show another app.
1118+
match shell_helper::screenshot_window(xid, pid) {
1119+
shell_helper::WindowCapture::Png(png) => return Ok(png),
1120+
shell_helper::WindowCapture::Unavailable => {
1121+
return Err(surface_identity_unproven(
1122+
xid,
1123+
"the GNOME helper could not capture this window (minimized, closed, \
1124+
or the screen is locked)",
1125+
))
1126+
}
1127+
shell_helper::WindowCapture::Unsupported => {}
1128+
}
1129+
}
11091130
screenshot_window_bytes_with_dispatch(
11101131
is_wayland(),
11111132
xid,
@@ -3490,6 +3511,11 @@ pub fn list_windows_dispatch(filter_pid: Option<u32>) -> Vec<WindowInfo> {
34903511
}
34913512
};
34923513
}
3514+
// A verified GNOME helper already supplies exact IDs and ownership.
3515+
// Do not probe unrelated compositor protocols for every preview frame.
3516+
if let Some(ws) = shell_helper::list_windows(filter_pid) {
3517+
return listed_windows(ws);
3518+
}
34933519
if let Some(ws) = kwin_helper::list_window_infos() {
34943520
return listed_windows(apply_pid_filter(ws, filter_pid));
34953521
}

0 commit comments

Comments
 (0)