Expose Semantics.identifier as Windows UIA AutomationId #161955

loic-peron-inetum-public · 2025-01-21T15:57:09Z

Expose Semantics.identifier as Windows AutomationId to ease application automation.

see #148763

followup to flutter/engine#53729 and flutter/engine#53476

Pre-launch Checklist

I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
I read the [Tree Hygiene] wiki page, which explains my responsibilities.
I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
I signed the [CLA].
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is [test-exempt].
I followed the [breaking change policy] and added [Data Driven Fixes] where supported.
All existing and new tests are passing.

chunhtai · 2025-02-05T22:40:15Z

Hi @loic-peron-inetum-public , are you still working on this pr?

loic-peron-inetum-public · 2025-02-06T10:18:16Z

Hi @loic-peron-inetum-public , are you still working on this pr?

Yes I am, but I have difficulties to find a Mac to fix the problems it created with the macOS desktop target, as well as time to work on the subject due to other pressing deadline.

The windows desktop changes have not been altered since last summer, and are used internally to automate tests on the project. They still seem to fulfill the intended purpose.

I intend to fix the tests as soon as possible, but I cannot make any promise.

Some concerns have been raised with enabling UIA support (flutter/engine#53476 (comment)). I have not been able to treat that and would need some guidance on this specific matter, as UIA seems mandatory for this feature: I have not been able to have AutomationId exposed without enabling UIA.

loic-sharma · 2025-03-04T21:59:28Z

I've converted this into a draft while it's still a work in progress. Feel free to mark this as ready for review once you're ready! :)

pgn001 · 2025-05-09T18:52:18Z

@loic-peron-inetum-public Hi, any update on this, as we also require the AutomationID.
Any suggestion to make it work locally until this gets merged/released?

loic-peron-inetum-public · 2025-06-19T14:52:41Z

@pgn001
First build the stock flutter engine according to https://github.com/flutter/flutter/tree/master/engine/src/flutter/docs/contributing
Then apply the patch for this PR and rebuild.
Finally use the custom flutter engine according to https://github.com/flutter/flutter/blob/master/docs/tool/README.md#using-a-locally-built-engine-with-the-flutter-tool

loic-peron-inetum-public · 2025-06-23T10:02:44Z

I've converted this into a draft while it's still a work in progress. Feel free to mark this as ready for review once you're ready! :)

I have updated the PR and removed the draft marker, and would like to have reviews from Flutter project members on this proposal.

engine/src/flutter/shell/platform/darwin/macos/framework/Source/FlutterEngineTest.mm

engine/src/flutter/shell/platform/windows/flutter_window.cc

engine/src/flutter/shell/platform/embedder/embedder.h

engine/src/flutter/shell/platform/windows/flutter_platform_node_delegate_windows.h

flutter-dashboard · 2025-07-28T13:30:38Z

This pull request executed golden file tests, but it has not been updated in a while (20+ days). Test results from Gold expire after as many days, so this pull request will need to be updated with a fresh commit in order to get results from Gold.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

engine/src/flutter/shell/platform/embedder/embedder.h

loic-sharma · 2025-08-04T19:56:08Z

engine/src/flutter/third_party/accessibility/ax/platform/ax_platform_node_win.cc

 }

+//
+// IAccessibleEx implementation.


Can you split this into two pull requests?

Add identifier to Flutter and pipe it to all embedders' accessibility bridge

Update Flutter Windows to implement IAccessibleEx, and use that to implement the AutomationId UIA property

I think the first one is very close to landing. However, for the second one I need to some research before I feel comfortable landing it. Since the bulk of the work is in that first pull request, I would land that first without blocking on the second pull request.

hellohuanlin · 2025-08-21T21:41:40Z

@loic-peron-inetum-public are you able to split it into 2 PRs as requested by @loic-sharma

loic-peron-inetum-public · 2025-08-22T05:13:50Z

@loic-peron-inetum-public are you able to split it into 2 PRs as requested by @loic-sharma

I will make time to rework this PR in September.

chunhtai · 2025-08-27T21:39:00Z

converted to draft, feel free to reopen when this is ready

flutter-dashboard · 2025-08-27T21:40:13Z

This pull request has been changed to a draft. The currently pending flutter-gold status will not be able to resolve until a new commit is pushed or the change is marked ready for review again.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

…icsNode2::identifier

…Bridge::SemanticsNode::identifier

…a::StringAttributes[HtmlId]

…ate::AuthorUniqueId

loic-peron-inetum-public · 2025-09-12T04:42:35Z

Rebased and added tests. Will split in two PR

one for common code (all but the last commit)
one specific fow Windows (the last commit)

loic-peron-inetum-public · 2025-09-16T08:30:11Z

Split in two parts:

[Desktop] Propagate SemanticsNode::identifier to AXPlatformNodeDelegate::AuthorUniqueId #175405 with common code
[Windows] Restore and enable IAccessibleEx implementation #175406 with Windows code

…te::AuthorUniqueId (#175405) Expose flutter Semantics.identifier as AXPlatformNodeDelegate::AuthorUniqueId. To be used for UI test automation through Windows AutomationId. see #148763 followup to #161955 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing.

…te::AuthorUniqueId (flutter#175405) Expose flutter Semantics.identifier as AXPlatformNodeDelegate::AuthorUniqueId. To be used for UI test automation through Windows AutomationId. see flutter#148763 followup to flutter#161955 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing.

github-actions bot added engine flutter/engine related. See also e: labels. a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) platform-windows Building on or for Windows specifically a: desktop Running on desktop labels Jan 21, 2025

loic-peron-inetum-public changed the title ~~expose Semantics.identifier as Windows UIA AutomationId~~ [WIP] expose Semantics.identifier as Windows UIA AutomationId Jan 22, 2025

chunhtai requested a review from loic-sharma February 5, 2025 22:39

loic-sharma marked this pull request as draft March 4, 2025 21:59

loic-peron-inetum-public force-pushed the windows-automationid branch from 677567b to d24a19e Compare March 13, 2025 14:22

github-actions bot added platform-linux Building on or for Linux specifically platform-macos labels Mar 13, 2025

loic-peron-inetum-public force-pushed the windows-automationid branch 2 times, most recently from 8ac9c57 to e037856 Compare March 14, 2025 06:35

loic-peron-inetum-public force-pushed the windows-automationid branch 2 times, most recently from ab142ac to 8d7525c Compare June 23, 2025 08:06

loic-peron-inetum-public marked this pull request as ready for review June 23, 2025 09:27

loic-peron-inetum-public requested review from cbracken and chinmaygarde as code owners June 23, 2025 09:27

loic-peron-inetum-public changed the title ~~[WIP] expose Semantics.identifier as Windows UIA AutomationId~~ Expose Semantics.identifier as Windows UIA AutomationId Jun 23, 2025