Repository navigation
[FEATURE]: Reliable, reusable GUI operation #32945
Description
Feature hasn't been suggested before.
- I have verified this feature I'm about to request hasn't been suggested before.
Describe the enhancement you want to request
A model can already click around a graphical app from the raw screen, but it does it weakly — re-reading pixels every step, guessing coordinates, and re-figuring out the same app on every run. It's slow, expensive, and brittle.
I'd like to make GUI operation solid. Where an app exposes an accessibility tree, the agent acts on real structure — controls, labels, values — with OCR to fill the gaps, instead of guessing from pixels: click, type, key, all through the existing permission flow.
And a successful run is recorded as a reusable flow — its states, locators, actions, and the checks that confirm each step — then replayed next time instead of re-explored. When the UI shifts and a check fails, only the broken step is repaired, not the whole flow.
Related to #20490, which covers the raw computer-use loop — screenshot, then inject mouse and keyboard, per OS. That's the pixel-based control this improves on: the focus here is reading structure instead of guessing coordinates, plus recording flows so a task isn't re-derived every run. The two can compose — raw input injection could sit underneath as one backend — but reliability and reuse are the point.
I have a working implementation. I'd rather agree on the shape here before a PR: how it surfaces as tools, where recorded flows live relative to skills, and the permission model for actions.
Activity
github-actions commentedon Jun 19, 2026
This issue might be a duplicate of existing issues. Please check:
If your proposal is specifically about the accessibility-tree-based approach and reusable recorded flows (on top of or distinct from #20490), please clarify how it differs and consider linking to that issue for context.
EtienneLescot commentedon Jun 20, 2026
Hi @rekram1-node — I drafted an RFC that tries to agree on shape first, per your note on this issue ("I'd rather agree on the shape here before a PR: how it surfaces as tools, where recorded flows live relative to skills, and the permission model for actions").
It's up as a draft PR: #33082 (docs only, adds
rfcs/computer-use.md). Cross-references this issue plus #20490 / #20917 / #26772 / #30755.A few places where I made calls and would value a sanity check before going further:
max_steps_per_task: 50budget, no per-action prompts (chose this over Codex's per-action model on UX grounds).computer_use.enabled: falseexperimental flag. You flagged on [FEATURE]: Computer Use / Desktop Control tool (screenshot + mouse/keyboard + browser automation) #20490 that you wanted this closed, which made me cautious — happy to flip to plugin-first if you'd rather see it land there and graduate.I'll wait on feedback here before pushing anything else. Happy to iterate on the doc.
github-actions commentedon Aug 20, 2026
To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.