feat(cua-computer): add experimental Windows/Linux computer-use fulfiller#112267
Conversation
openclaw-barnacle
Bot
added
docs
github-actions
Bot
added
the
dependencies-changed
Dependency GuardThis PR changes dependency-related files. Maintainers should confirm these changes are intentional. Changed files:
Maintainer follow-up:
|
Dependency graph changes notedThis PR includes dependency graph changes. The dependency guard is informational because the PR author is a repository admin or a member of
Security review is still recommended before merge when the dependency graph change is intentional. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: feat(cua-computer): add experimental Windows/Linux computer-use fulfiller This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
…ller Bundled plugin that fulfills the capability-based computer.act + screen.snapshot node contract on Windows and Linux by supervising a pinned cua-driver 0.10.x daemon over MCP stdio. macOS keeps the Peekaboo fulfiller; this plugin is disabled by default and never available on darwin. Grounded in cua-driver 0.10.0 source (tool schemas, refusal codes, coordinate spaces, session/daemon lifecycle). Notable safety and correctness properties: - Deny-by-default env allowlist so OpenClaw secrets (provider/channel tokens, CUA_API_KEY) never reach the separately installed daemon; telemetry and update checks forced off. - Version-gated handshake (exact-minor pin + capability/schema version), time-bounded so a corrected driver recovers without a node restart. - Robust daemon supervision: full readiness-budget polling, startup-race tolerance, signal-death and spawn-error recovery, shared-daemon lifecycle (never killed on dispose). - Frame authorization preserved within upstream limits (generation + full live geometry; capture refused when screen and screenshot geometry diverge). - Action mapping refuses inputs cua-driver cannot faithfully deliver: layout-shifted keys, modifier-held drag/scroll, Linux modifier clicks, hold_key/mouse down-up, non-positive scroll; drag duration clamped.
99b9cab to
76a36ed
Compare
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: feat(cua-computer): add experimental Windows/Linux computer-use fulfiller This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
|
Land-ready verification (maintainer-directed landing): Work done: first cross-platform (Windows/Linux) computer-use fulfiller as the bundled Local proof (trusted source, focused):
Hosted proof: full CI green on head Known proof gaps:
|
|
Merged via squash.
|
…ller (openclaw#112267) * feat(cua-computer): add experimental Windows/Linux computer-use fulfiller Bundled plugin that fulfills the capability-based computer.act + screen.snapshot node contract on Windows and Linux by supervising a pinned cua-driver 0.10.x daemon over MCP stdio. macOS keeps the Peekaboo fulfiller; this plugin is disabled by default and never available on darwin. Grounded in cua-driver 0.10.0 source (tool schemas, refusal codes, coordinate spaces, session/daemon lifecycle). Notable safety and correctness properties: - Deny-by-default env allowlist so OpenClaw secrets (provider/channel tokens, CUA_API_KEY) never reach the separately installed daemon; telemetry and update checks forced off. - Version-gated handshake (exact-minor pin + capability/schema version), time-bounded so a corrected driver recovers without a node restart. - Robust daemon supervision: full readiness-budget polling, startup-race tolerance, signal-death and spawn-error recovery, shared-daemon lifecycle (never killed on dispose). - Frame authorization preserved within upstream limits (generation + full live geometry; capture refused when screen and screenshot geometry diverge). - Action mapping refuses inputs cua-driver cannot faithfully deliver: layout-shifted keys, modifier-held drag/scroll, Linux modifier clicks, hold_key/mouse down-up, non-positive scroll; drag duration clamped. * fix(cua-computer): satisfy lint, test-types, dead-code, and docs-map gates
1 participant
What Problem This Solves
OpenClaw's computer use has a platform-neutral contract (capability-based since #112107) but only one fulfiller: the macOS app's embedded Peekaboo service. Windows and Linux desktops cannot be driven at all. This PR adds the first cross-platform fulfiller as a bundled plugin,
cua-computer, backed by trycua/cua'scua-driverdaemon (Rust; UIA on Windows, X11/AT-SPI on Linux).Why This Change Was Made
Design follows the repo's owner-boundary rules: core stays cua-agnostic. The plugin registers
screen.snapshot+computer.actnode-host commands (the linux-canvas pattern) so a headlessopenclaw node runhost in a Windows/Linux desktop session advertises them; gateway pairing/arming policy already accepts desktop platforms declaring these since #112107. Implementation notes, all grounded in cua-driver 0.10.0 source (contract extraction with file:line citations in the PR discussion):cua-driver mcpchild; spawnscua-driver servefirst on Linux/Windows (upstream only auto-starts the daemon on macOS). Telemetry (CUA_DRIVER_RS_TELEMETRY_ENABLED) and update checks are forced off for OpenClaw-managed processes.capability_version/schema_version"1" at handshake; anything else refuses with a typed error. Upstream is prerelease with fast churn — the pin turns churn into a visible refusal, not silent breakage.displayFrameIdis synthesized (geometry + daemon-generation hash) because upstream exposes no stable display identity; coordinate actions verify the echoed token AND re-check live geometry viaget_screen_sizebefore injecting, preserving the frame-binding invariant. Screenshots are downscaled to the reference width via rastermill (plugin-local dep) so delivered dimensions equal the coordinate space.hold_key/left_mouse_down/left_mouse_upunsupported (upstream has no desktop-scope hold API); macOS never available (Peekaboo owns it); native Wayland only via upstream's own opt-in env, KDE unsupported upstream.User Impact
New opt-in capability (plugin
enabledByDefault: false): with cua-driver 0.10.x installed and the plugin enabled, a paired Windows or Linux X11 node can be driven by thecomputertool under the exact same pairing, arming, frame-binding, and vision-model gates as macOS. Nothing changes for users who don't enable it.Evidence
node scripts/run-vitest.mjs extensions/cua-computer(mapping tables, frame staleness/geometry drift, refusal codes, version gate, serialization, availability gating) plus unchanged greencomputer-toolandnode-command-policysuites.cua-driver-rs-v0.10.0with per-claim file:line citations.Review hardening
This PR went through an extended adversarial autoreview (13 rounds). Two genuine P1 security issues were caught and fixed before land: the initial code copied the entire process environment into the daemon (secret leak), and an interim allowlist fix was itself too broad (
CUA_prefix would forwardCUA_API_KEY). Both are now closed with a deny-by-default exact allowlist plus a regression test. Several coordinate-space and delivery-mode findings were verified against cua-driver 0.10 source and rejected as source-contradicted (desktop scope is global/foreground and ignoresdelivery_mode;press_keyapplies modifiers on Wayland by internal hotkey promotion; screen and screenshot geometry are equal on all supported backends) — with clarifying invariant comments added so they are not re-flagged.