Add accessibility identifier to `SemanticsProperties` #138331

bartekpacia · 2023-11-12T21:58:58Z

This PR adds String? identifier to Semantics and SemanticsProperties. The identifier will be exposed on Android as resource-id and on iOS as accessibilityIdentifier.

Mainly targeted at #17988

Initial Engine PR with Android support: flutter/engine#47961
iOS Engine PR: flutter/engine#48858

Migration

This change breaks the SemanticsUpdateBuilder API which is on the Framework<-->Engine border. For more details see engine PR.

Steps:
part 1: [engine] add SemanticsUpdateBuilderNew flutter/engine#47961
part 2: [flutter] use SemanticsUpdateBuilderNew <-- we are here
part 3: [engine] update SemanticsUpdateBuilder to be the same as SemanticsUpdateBuilderNew flutter/engine#48882
part 4: [flutter] use (now updated) SemanticsUpdateBuilder again #139942
part 5: [engine] remove SemanticsBuilderNew flutter/engine#49139

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

chunhtai

which accessibilitynodeinfo property would this new identifier set to? There doesn't seem to be a resourceId in the accessibilitynodeinfo https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo

chunhtai

ah it is viewIdResourceName just saw the engine pr

packages/flutter/lib/src/rendering/proxy_box.dart

packages/flutter/lib/src/semantics/semantics.dart

chunhtai · 2023-11-17T18:07:33Z

packages/flutter/lib/src/semantics/semantics.dart

This doc seems a bit confusing. The UIAutomator interact with semantics node, not renderObject.

Good catch, thank you. I'm still new to this code. Will fix.

Here I took inspiration from other SemanticsConfiguration fields, e.g.:

https://github.com/flutter/flutter/blob/0b98178af46c82d48dd6e008290e23f8a21f122a/packages/flutter/lib/src/semantics/semantics.dart#L4231-L4240

What'd you change?

packages/flutter/lib/src/semantics/semantics.dart

chunhtai

LGTM, just some nit picking

packages/flutter/lib/src/semantics/semantics.dart

bartekpacia · 2023-12-01T19:30:12Z

Thank you - I applied your suggestions.

…ndroid (#47961) Accompanying framework PR: flutter/flutter#138331 This PR implements support for exposing `SemanticsProperties.identifier` on Android as `resource-id`. Mainly targeted at flutter/flutter#17988. Would also fix https://github.com/flutter/flutter/issues/issues/137735 but it was marked as duplicate. Anyway, there's a lot of context in that issue. This PR requires changing the `SemanticsUpdateBuilder` interface (defined in engine) that framework depends on, so it requires introducing a temporary API ([see question I asked on Discord](https://discord.com/channels/608014603317936148/608018585025118217/1174845658033819729) to learn more about this approach). Steps: **part 1: [engine] add `SemanticsUpdateBuilderNew`** <-- we are here part 2: [flutter] use `SemanticsUpdateBuilderNew` part 3: [engine] update `SemanticsUpdateBuilder` to be the same as `SemanticsUpdateBuilderNew`* part 4: [flutter] use (now updated) `SemanticsUpdateBuilder` again. part 5: [engine] remove `SemanticsBuilderNew` I'd like to do these changes first, and only then continue with [the proper framework PR](flutter/flutter#138331). *More specifically: update `SemanticsUpdateBuilder.updateNode()` to be the same as `SemanticsUpdateBuilderNew.updateNode()`. Number of arguments that function takes is the only change. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

srawlins · 2023-12-09T23:05:43Z

@bartekpacia the latest "Linux customer_testing" bot failure says it grabbed devtools commit 914356dedc97b4bafc33cc3f9c0334fefbb3417f. Not sure why...

bartekpacia · 2023-12-09T23:38:59Z

Yeah, indeed. At first I thought the commit is maybe pinned elsewhere but nope.

…guration

…ngine

Co-authored-by: chunhtai <[email protected]>

…r/flutter#138331)

This PR adds `String? identifier` to `SemanticsUpdateBuilder` (currently it's only available in the temproary `SemanticsUpdateBuilderNew` API. This is mainly targeted at flutter/flutter#17988 Steps: part 1: [engine] add `SemanticsUpdateBuilderNew` #47961 part 2: [flutter] use `SemanticsUpdateBuilderNew` flutter/flutter#138331 **part 3: [engine] update `SemanticsUpdateBuilder` to be the same as `SemanticsUpdateBuilderNew`** <-- we are here part 4: [flutter] use (now updated) `SemanticsUpdateBuilder` again. part 5: [engine] remove `SemanticsBuilderNew`

Manual roll requested by [email protected] flutter/flutter@c642f4e...9719097 2023-12-11 [email protected] Roll Flutter Engine from 5587d26aa2d4 to 4c309195b79d (1 revision) (flutter/flutter#139936) 2023-12-11 [email protected] Fix SelectionArea select-word edge cases (flutter/flutter#136920) 2023-12-11 [email protected] Roll Flutter Engine from 9b85b76db0de to 5587d26aa2d4 (3 revisions) (flutter/flutter#139933) 2023-12-11 [email protected] Roll Flutter Engine from 7eb6b7cab60c to 9b85b76db0de (2 revisions) (flutter/flutter#139931) 2023-12-11 [email protected] Use dart analyze package for `num.clamp` (flutter/flutter#139867) 2023-12-11 [email protected] Roll pub packages (flutter/flutter#139926) 2023-12-11 [email protected] Handle the case when _CupertinoBackGestureDetector is disposed during the drag. (flutter/flutter#139585) 2023-12-11 [email protected] Add accessibility identifier to `SemanticsProperties` (flutter/flutter#138331) 2023-12-11 [email protected] Improve slider's value indicator display test (flutter/flutter#139198) 2023-12-11 [email protected] Add `enabled` property to `ExpansionTile` (flutter/flutter#139519) 2023-12-11 [email protected] Roll Packages from 6cd0657 to cb6dbcd (9 revisions) (flutter/flutter#139911) 2023-12-11 [email protected] Roll Flutter Engine from bc0222b64c96 to 7eb6b7cab60c (1 revision) (flutter/flutter#139891) 2023-12-10 [email protected] Roll Flutter Engine from fb80aafd259b to bc0222b64c96 (1 revision) (flutter/flutter#139885) 2023-12-09 [email protected] Roll pub packages (flutter/flutter#139864) 2023-12-09 [email protected] Roll Flutter Engine from b75960a5820a to fb80aafd259b (1 revision) (flutter/flutter#139863) 2023-12-09 [email protected] Roll Flutter Engine from e80c090d09c6 to b75960a5820a (1 revision) (flutter/flutter#139853) 2023-12-09 [email protected] Roll Flutter Engine from 101396fd3b82 to e80c090d09c6 (2 revisions) (flutter/flutter#139851) 2023-12-09 [email protected] Roll Flutter Engine from 503584615fd7 to 101396fd3b82 (2 revisions) (flutter/flutter#139847) 2023-12-09 [email protected] Roll Flutter Engine from e9cb19fa637a to 503584615fd7 (1 revision) (flutter/flutter#139837) 2023-12-08 [email protected] Roll Flutter Engine from 7dc51b85a634 to e9cb19fa637a (1 revision) (flutter/flutter#139831) 2023-12-08 [email protected] [flutter release] Add cherry pick template for pull request description (flutter/flutter#139590) 2023-12-08 [email protected] Roll Flutter Engine from 03c5f016e919 to 7dc51b85a634 (1 revision) (flutter/flutter#139829) 2023-12-08 [email protected] Add Overlay.wrap for convenience (flutter/flutter#139823) 2023-12-08 [email protected] Roll Flutter Engine from 72da960e2ef2 to 03c5f016e919 (1 revision) (flutter/flutter#139826) 2023-12-08 [email protected] Roll Flutter Engine from 5dd2619c282b to 72da960e2ef2 (1 revision) (flutter/flutter#139821) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…y `SemanticsUpdateBuilderNew` (#139942) This PR removes all usages of the temporary `SemanticsUpdateBuilderNew` API in favor of `SemanticsUpdateBuilder`. These two APIs are the same as of now. This is mainly targeted at #17988 Steps: part 1: [engine] add `SemanticsUpdateBuilderNew` flutter/engine#47961 part 2: [flutter] use `SemanticsUpdateBuilderNew` #138331 part 3: [engine] update `SemanticsUpdateBuilder` to be the same as `SemanticsUpdateBuilderNew` flutter/engine#48882 **part 4: [flutter] use (now updated) `SemanticsUpdateBuilder` again** <-- we are here part 5: [engine] remove `SemanticsBuilderNew`

) This PR completest the migration by removing `SemanticsUpdateBuilderNew` from the engine. This is mainly targeted at flutter/flutter#17988 Steps: part 1: [engine] add `SemanticsUpdateBuilderNew` #47961 part 2: [flutter] use `SemanticsUpdateBuilderNew` flutter/flutter#138331 part 3: [engine] update `SemanticsUpdateBuilder` to be the same as `SemanticsUpdateBuilderNew` #48882 part 4: [flutter] use (now updated) `SemanticsUpdateBuilder` again flutter/flutter#139942 **part 5: [engine] remove `SemanticsBuilderNew`** <-- we are here

…r/flutter#138331)

github-actions bot added framework flutter/packages/flutter repository. See also f: labels. a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) labels Nov 12, 2023

bartekpacia mentioned this pull request Nov 12, 2023

Add support for exposing accessibility identifier as resource-id on Android flutter/engine#47961

Merged

8 tasks

goderbauer requested a review from chunhtai November 14, 2023 23:15

chunhtai reviewed Nov 17, 2023

View reviewed changes

bartekpacia mentioned this pull request Nov 18, 2023

[Feature Request] Add support for flutter element keys mobile-dev-inc/Maestro#1424

Closed

bartekpacia requested a review from chunhtai November 27, 2023 21:45

yjbanov reviewed Nov 30, 2023

View reviewed changes

packages/flutter/lib/src/semantics/semantics.dart Outdated Show resolved Hide resolved

packages/flutter/lib/src/semantics/semantics.dart Outdated Show resolved Hide resolved

chunhtai reviewed Dec 1, 2023

View reviewed changes

packages/flutter/lib/src/semantics/semantics.dart Outdated Show resolved Hide resolved

packages/flutter/lib/src/semantics/semantics.dart Outdated Show resolved Hide resolved

packages/flutter/lib/src/semantics/semantics.dart Outdated Show resolved Hide resolved

chunhtai approved these changes Dec 1, 2023

View reviewed changes

bartekpacia marked this pull request as ready for review December 5, 2023 08:41

github-actions bot added the a: tests "flutter test", flutter_test, or one of our tests label Dec 6, 2023

bartekpacia mentioned this pull request Dec 6, 2023

Re-enable a skipped test in inspector_service_test flutter/devtools#6902

Closed

kenzieschmoll mentioned this pull request Dec 6, 2023

inspector_service_test.dart: disable "widget tree" test flutter/devtools#6903

Merged

7 tasks

bartekpacia mentioned this pull request Dec 9, 2023

Bump flutter_devtools to c8af1c0cdb77142f7ad0236b074e7c068498da0b flutter/tests#322

Merged

8 tasks

bartekpacia and others added 9 commits December 11, 2023 15:23

add identifier field to SemanticsProperties and SemanticsConfiguration

fb0f296

add identifier to RenderSemanticsAnnotations.describeSemanticsConfi…

e85552e

…guration

semantics: add support for exposing accessibility identifier to the e…

1f13a3a

…ngine

SemanticsNode: add identifier to debugFillProperties()

8c20e07

fix bug in SemanticsConfiguration.absorb for identifier

c07d779

SemanticsConfiguration.copy: fix duplicated identifier assignment

33bcea2

improve comment in SemanticsProperties.identifier

13193d5

use doc templates

93e4595

Update packages/flutter/lib/src/semantics/semantics.dart

bf7aa86

Co-authored-by: chunhtai <[email protected]>

trigger CI

a2083fb

bartekpacia added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 11, 2023

auto-submit bot merged commit 948523b into flutter:master Dec 11, 2023

bartekpacia deleted the feature/accessibility_identifier branch December 11, 2023 18:04

engine-flutter-autoroll mentioned this pull request Dec 11, 2023

Roll Flutter from c642f4e012f4 to 948523b80c9c (18 revisions) flutter/packages#5645

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 11, 2023

948523b Add accessibility identifier to SemanticsProperties (flutte…

95dc5a3

…r/flutter#138331)

bartekpacia mentioned this pull request Dec 11, 2023

SemanticsUpdateBuilder migration: introduce identifier flutter/engine#48882

Merged

8 tasks

engine-flutter-autoroll mentioned this pull request Dec 11, 2023

Roll Flutter from c642f4e012f4 to 35c52e7beedc (19 revisions) flutter/packages#5646

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 11, 2023

948523b Add accessibility identifier to SemanticsProperties (flutte…

51b75e8

…r/flutter#138331)

engine-flutter-autoroll mentioned this pull request Dec 11, 2023

Roll Flutter from c642f4e012f4 to 44beb843aa79 (20 revisions) flutter/packages#5647

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 11, 2023

948523b Add accessibility identifier to SemanticsProperties (flutte…

b416ad5

…r/flutter#138331)

engine-flutter-autoroll mentioned this pull request Dec 11, 2023

Manual roll Flutter from c642f4e012f4 to 97190973d6f5 (25 revisions) flutter/packages#5648

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 11, 2023

948523b Add accessibility identifier to SemanticsProperties (flutte…

2906bd4

…r/flutter#138331)

bartekpacia mentioned this pull request Dec 11, 2023

Use SemanticsUpdateBuilder again, remove all references to temporary SemanticsUpdateBuilderNew #139942

Merged

8 tasks

bartekpacia mentioned this pull request Dec 16, 2023

Delete SemanticsUpdateBuilderNew and all references and usages flutter/engine#49139

Merged

8 tasks

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 16, 2024

948523b Add accessibility identifier to SemanticsProperties (flutte…

90834a5

…r/flutter#138331)

huycozy mentioned this pull request Feb 29, 2024

TextField Acessibility ID not persist after build #144357

Closed

nicerloop mentioned this pull request May 21, 2024

[Windows] Missing automationId for testing purposes #148763

Open

herby-naidz mentioned this pull request Nov 26, 2024

Is the readme file correct about finding by key appium/appium-flutter-driver#754

Closed

KazuCocoa mentioned this pull request Dec 3, 2025

Flutter TextField does not expose resource-id or stable element identifier in Appium Inspector appium/appium-flutter-driver#835

Open

Add accessibility identifier to SemanticsProperties #138331

Add accessibility identifier to SemanticsProperties #138331

Uh oh!

Conversation

bartekpacia commented Nov 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration

Pre-launch Checklist

Uh oh!

chunhtai left a comment

Choose a reason for hiding this comment

Uh oh!

chunhtai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chunhtai Nov 17, 2023

Choose a reason for hiding this comment

Uh oh!

bartekpacia Nov 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bartekpacia Nov 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chunhtai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bartekpacia commented Dec 1, 2023

Uh oh!

srawlins commented Dec 9, 2023

Uh oh!

bartekpacia commented Dec 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add accessibility identifier to `SemanticsProperties` #138331

Add accessibility identifier to `SemanticsProperties` #138331

bartekpacia commented Nov 12, 2023 •

edited

Loading

bartekpacia Nov 17, 2023 •

edited

Loading