Skip to content

fix(cua-driver): decode GNOME window snapshots through typed D-Bus - #4809

Closed
louistiti wants to merge 2 commits into
trycua:mainfrom
louistiti:fix/gnome-window-snapshot-decoding
Closed

louistiti wants to merge 2 commits into
trycua:mainfrom
louistiti:fix/gnome-window-snapshot-decoding

Conversation

@louistiti

@louistiti louistiti commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem this solves: an apostrophe or escaped quotation mark in any GNOME window title can prevent the driver from parsing the complete WinRects window snapshot. Discovery then loses authoritative IDs and geometry for unrelated windows too, and window capture or activation can fail even though full-desktop capture still works.
  • What changed: read GetRects as its declared D-Bus string using the existing zbus dependency. Parse the decoded JSON directly for listing, geometry, trusted window lookup and focus verification. Preserve the verified unique helper owner, browser-helper version checks and bounded call behavior. Declare D-Bus as a native test dependency in the Nix unit-check derivation so the private-session transport test can execute in its sandbox.

Reproduction and user impact

On GNOME/Wayland with the WinRects helper active:

  1. Keep a terminal window open and set its title to Tomorrow's meeting.
  2. Open another application, such as Feishu, and ask the driver to list its windows, capture its window or activate it.
  3. GetRects still returns both windows, but gdbus call renders the JSON inside a GVariant string with an additional escaping layer. The existing first-[/last-] extraction leaves that escaping intact, so serde_json rejects the entire snapshot before PID filtering.
  4. Rename the terminal to Plain title and repeat: the parser succeeds.

This pattern was observed in a real GNOME session: punctuation in a terminal title broke discovery and window targeting for an unrelated Feishu window. Titles and IDs above are synthetic. No private session data is included.

A headless reproduction using GLib-generated GVariant replies and the actual upstream Rust parse_window_origin function confirms the bug in both cua-driver-rs-v0.34.0 and current main:

Title Expected origin Actual origin
Plain title (14, 12) (14, 12)
Tomorrow's meeting (14, 12) None
A "quoted" title (14, 12) None
飞书 (14, 12) (14, 12)

The replacement regression uses a mock D-Bus service and checks that punctuation, backslashes, newlines and Unicode survive the transport, and that a separate target window can still be filtered and located.

Before / after illustration

These AI-generated diagrams explain the reproduced parser behavior. They are not desktop screenshots or evidence of a live Feishu replay. The desktop stays visible in both; only authoritative window metadata becomes unavailable/available. Assets live on a separate fork branch and are not part of the source diff.

Before Proposed fix
Quoted title breaks the complete snapshot Typed D-Bus preserves both window records

Related work

No duplicate parsing fix was found in the reviewed open and closed PRs. #2995 changes GNOME compatibility/geometry but retains display-text parsing; #4011 changes cursor dispatch; #3887 addresses fractional scaling. This PR does not resolve those separate issues.

RFC: not required; this repairs decoding of an existing native D-Bus reply without changing a public SDK, CLI, MCP, helper protocol, compatibility policy or permission boundary. This is a small, unselected contribution pending maintainer review.

Compatibility and risk

  • User-visible, API/CLI/MCP, migration, permission, or platform impact: scoped to the Linux GNOME helper path used for Wayland discovery and targeting. macOS, Windows, X11 and other compositor adapters are not modified. No new dependency or helper installation is needed. The wire method remains GetRects() -> string containing JSON.
  • Risk and rollback: replaces the snapshot transport, so helper connection failure, timeout, exact owner identity and synchronous calls from an existing Tokio runtime need validation. The separate-thread runtime follows the existing KDE adapter pattern; the snapshot operation retains an 800 ms timeout. Revert this file to roll back.

Validation

Candidate: 7dc974445297fd0408878efe520651c963cc6670, based on main a7524cfd1d3e959963b43954d43f27c4bd260f08. Rust 1.97.1; no dependency changes.

The original Rust checks below ran on 613e10ea97d637de9e16af20f2f4d53b2c5b326f. Commit 69476a44bdcc69512f05806c9e7f08b10244025d corrected only the committer identity/signature and retained its tree. The current candidate adds only nativeCheckInputs = [ pkgs.dbus ]; and its explanatory comment in the Nix unit-test derivation; all Rust sources remain identical. GitHub verifies both follow-up signatures as valid.

  • cargo fmt --all -- --check and git diff --check: passed.
  • Focused cargo test --locked -p platform-linux --lib wayland::shell_helper::tests -- --nocapture: eight ordinary tests passed; the private-bus child regression also passed.
  • cargo test --locked -p platform-linux --lib: 637 passed, one failed, 11 ignored. The normal private-bus regression passed and invokes its ignored transport owner test under dbus-run-session; CI's whole-crate test command therefore exercises the regression without a desktop bus or extra runner filter.
  • cargo test --locked -p platform-linux --all-targets -- --test-threads=1: the same unchanged overlay::tests::unreported_arrival_releases_the_waiter_after_the_cap assertion failed at overlay.rs:3853. The overlay test also failed when rerun alone with --exact. Cargo stopped after the library suite, so subsequent targets are not claimed as passing.
  • Strict cargo clippy --locked -p platform-linux --lib --tests --no-deps -- -D warnings: failed on six distinct diagnostics in unchanged tools/impl_.rs, input/mod.rs, wayland/mod.rs and wayland/hyprland.rs (11 emitted diagnostics across targets). None points to the changed helper. No clean lint pass is claimed.
  • Manual/platform evidence: the actual upstream parser rejects both GLib-generated quoted-title fixtures while accepting normal-title controls. The isolated D-Bus regression verifies the decoded string, exact title, focus, geometry and an unrelated target PID. This proves the transport/parser contract, not delivery to a live application.
  • CI on 69476a4: the ordinary Linux Rust job passed, including typed_window_snapshot_runs_on_private_bus. The Nix unit job stopped before decoder assertions because dbus-run-session was absent (637 passed, 1 failed, 11 ignored). The previously observed local overlay failure passed in CI. The current candidate supplies the missing executable as a check-only dependency and keeps the transport regression enabled.
  • Follow-up validation on 7dc9744: nixfmt --check nix/cua-driver/tests/rust-unit.nix, cargo fmt --all -- --check, and git diff --check passed. Nix is unavailable locally. The new Nix CI run is awaiting upstream maintainer approval for the fork update; no successful Nix result is claimed yet.
  • Known gaps: ordinary PR CI, the affected live GNOME lane and the required representative desktop matrix must run on the final candidate before marking ready or merging. No live Feishu replay or cross-platform desktop certification is claimed. The overlay failure and strict-lint diagnostics in unchanged files remain visible rather than expanding this one-file fix.

Contributor and release checks

  • The PR is focused and the description matches the helper change and its Nix test dependency.
  • This change does not require an RFC.
  • Tests and platform-evidence gaps are explained.
  • The PR title is a Conventional Commit describing the production change.
  • No external contribution was transplanted; implementation follows existing repository SDK facilities.
  • Commit authorship uses the contributor's GitHub-linked email and verified signing key.
  • Before merge: required CI, desktop validation and CLA checks pass.
  • Release: user-visible bug fix; a patch release is intended, so no-release does not apply.

Agent assistance was used for investigation, drafting and validation. The human submitter remains responsible for reviewing the diff and the evidence.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@louistiti
louistiti force-pushed the fix/gnome-window-snapshot-decoding branch from 613e10e to 69476a4 Compare October 7, 2026 08:46
@louistiti

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Oct 7, 2026
@louistiti
louistiti marked this pull request as ready for review October 7, 2026 08:50
@f-trycua

f-trycua commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR. I've labelled and assigned it for review. CI and the CLA check will run on the branch; please sign the CLA if the bot asks, and we'll follow up with review comments.

@louistiti
louistiti requested a review from ddupont808 as a code owner October 8, 2026 08:29
cursor Bot pushed a commit that referenced this pull request Oct 8, 2026
Parse the typed GetRects reply as raw JSON instead of slicing a bracket
span, and reuse that parser for focus. Salvaged from #4809, including
the malformed-snapshot fixtures and the Nix dbus check input.

Co-authored-by: injaneity <[email protected]>
f-trycua added a commit that referenced this pull request Oct 8, 2026
…er (#4879)

* perf(cua-driver): reuse one session-bus connection for the GNOME helper

Every WinRects call spawned a gdbus subprocess, and owner attestation
alone makes three D-Bus calls before the real one. A live preview pays
that on every frame. Replace the subprocess with one zbus connection on
a dedicated single-thread Tokio runtime, so callers inside or outside
Tokio can block on it without nesting runtimes.

Bodies are typed on the wire, and replies keep the textual shape the
existing parsers read (GetRects now arrives as raw JSON, which also
avoids gdbus quoting). A timeout or transport error drops the
connection and the next call reconnects; a mutation is never replayed.
An error reply, such as no helper on the bus, keeps the connection.

T3 Code measured window capture going from 381 ms to 58 ms with this
transport. Ported from their Linux patch to Cua Driver 0.34.0
(pingdotgg/t3code#16975, MIT, Copyright (c) 2026 T3 Tools Inc.); the
reply-versus-transport split is new here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(cua-driver): cover GetRects titles gdbus would have escaped

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(cua-driver): strict GetRects parsing and run the bus test in Nix

Parse the typed GetRects reply as raw JSON instead of slicing a bracket
span, and reuse that parser for focus. Salvaged from #4809, including
the malformed-snapshot fixtures and the Nix dbus check input.

Co-authored-by: injaneity <[email protected]>

* test(cua-driver): start the private test bus from its own config

dbus-daemon --session needs the system session.conf, which the Nix
build sandbox lacks; the daemon then exits without an address. Write a
minimal session config listening in a temp dir instead.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Louis Grenard <[email protected]>

Copy link
Copy Markdown
Collaborator

Thanks Louis, this was a great catch. Your strict GetRects parsing, the malformed-snapshot test and the Nix dbus check input all landed in #4879 (merged in 48cc491), credited to you. Closing this one as superseded.

@injaneity injaneity closed this Oct 8, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants