Add set semantics enabled API and wire iOS a11y bridge #161265

chunhtai · 2025-01-07T21:37:02Z

previously the only correct way to enable semantics is that ios embedding receive signal from native OS, it call SetSemanticsEnabled to shell and then to dart to enable semantics tree generation.

If for some reason framework decide to enable semantics first, e.g. through SemanticsBinding.instance.ensureSemantics(typically due to integration test or ci that wants to test semantics), the update will be dropped in shell. Even if it later on receive signal from native OS to turn on semantics, it can't construct the complete accessibility tree in embedding because the updatesemantics sends diff update and previous updates are gone forever. It will end up in a broken state.

This pr changes so that the only source of truth will be in the framework side. When framework starts generating the the semantics tree, it will call SetSemanticsTreeEnabled through dart:ui, and the embedding needs to prepare itself to accept semantics update after receiving the message.

This however require some refactoring on iOS embedding because it will only create a11y bridge when receiving OS notification when assitive technologies turns on.

This requires three phase transition

add an empty dart:ui API setSemanticsTreeEnabled
makes framework calls the empty API.
merge this pr with actual implementation of setSemanticsTreeEnabled

I will do the android part in a separate pr

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
I followed the breaking change policy and added Data Driven Fixes where supported.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

engine/src/flutter/runtime/runtime_controller.h

cbracken

@chunhtai can you add a PR description with the rationale for the change, related bug(s), etc.?

chunhtai · 2025-01-13T18:07:43Z

@chunhtai can you add a PR description with the rationale for the change, related bug(s), etc.?

sorry, forgot to add description. Done

goderbauer

LGTM for dart:ui (after doc nits are resolved)

goderbauer · 2025-02-03T23:56:18Z

engine/src/flutter/lib/ui/platform_dispatcher.dart

+  /// Informs the engine whether the framework is generating a semantics tree.
+  ///
+  /// Only framework knows when semantics tree should be generated. It uses this
+  /// method to notify the engine such event.


Maybe better:

Suggested change

/// method to notify the engine such event.

/// method to notify the engine whether the framework will generate a semantics tree.

engine/src/flutter/lib/ui/platform_dispatcher.dart

engine/src/flutter/lib/ui/window.dart

justinmc · 2025-02-26T00:05:00Z

@chunhtai heads up there are failures here if you hadn't seen.

cbracken

Just a few small nits then good to go!

engine/src/flutter/runtime/runtime_controller.h

engine/src/flutter/runtime/runtime_controller_unittests.cc

engine/src/flutter/shell/platform/darwin/ios/platform_view_ios.mm

engine/src/flutter/shell/platform/darwin/ios/platform_view_ios_test.mm

cbracken

auto-submit · 2025-03-18T23:07:17Z

autosubmit label was removed for flutter/flutter/161265, because - The status or check suite Mac_arm64 module_test_ios has failed. Please fix the issues identified (or deflake) before re-applying this label.

…er/flutter#161265)

fixes flutter#158399 old pr flutter/engine#56691 previously the only correct way to enable semantics is that ios embedding receive signal from native OS, it call SetSemanticsEnabled to shell and then to dart to enable semantics tree generation. If for some reason framework decide to enable semantics first, e.g. through SemanticsBinding.instance.ensureSemantics(typically due to integration test or ci that wants to test semantics), the update will be dropped in shell. Even if it later on receive signal from native OS to turn on semantics, it can't construct the complete accessibility tree in embedding because the updatesemantics sends diff update and previous updates are gone forever. It will end up in a broken state. This pr changes so that the only source of truth will be in the framework side. When framework starts generating the the semantics tree, it will call SetSemanticsTreeEnabled through dart:ui, and the embedding needs to prepare itself to accept semantics update after receiving the message. This however require some refactoring on iOS embedding because it will only create a11y bridge when receiving OS notification when assitive technologies turns on. This requires three phase transition add an empty dart:ui API setSemanticsTreeEnabled makes framework calls the empty API. merge this pr with actual implementation of setSemanticsTreeEnabled I will do the android part in a separate pr ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord].  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

…ter#161265)" This reverts commit cc04ca4.

fixes flutter#158399 old pr flutter/engine#56691 previously the only correct way to enable semantics is that ios embedding receive signal from native OS, it call SetSemanticsEnabled to shell and then to dart to enable semantics tree generation. If for some reason framework decide to enable semantics first, e.g. through SemanticsBinding.instance.ensureSemantics(typically due to integration test or ci that wants to test semantics), the update will be dropped in shell. Even if it later on receive signal from native OS to turn on semantics, it can't construct the complete accessibility tree in embedding because the updatesemantics sends diff update and previous updates are gone forever. It will end up in a broken state. This pr changes so that the only source of truth will be in the framework side. When framework starts generating the the semantics tree, it will call SetSemanticsTreeEnabled through dart:ui, and the embedding needs to prepare itself to accept semantics update after receiving the message. This however require some refactoring on iOS embedding because it will only create a11y bridge when receiving OS notification when assitive technologies turns on. This requires three phase transition add an empty dart:ui API setSemanticsTreeEnabled makes framework calls the empty API. merge this pr with actual implementation of setSemanticsTreeEnabled I will do the android part in a separate pr ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord].  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

…ter#161265)" This reverts commit cc04ca4.

github-actions bot added platform-ios iOS applications specifically framework flutter/packages/flutter repository. See also f: labels. engine flutter/engine related. See also e: labels. platform-web Web applications specifically labels Jan 7, 2025

chunhtai mentioned this pull request Jan 8, 2025

Google test doesn't support engine changes #161323

Closed

chunhtai requested review from cbracken and goderbauer January 8, 2025 17:14

chunhtai force-pushed the issues/158399-r3 branch from 53c5a6a to c601cac Compare January 8, 2025 17:58

github-actions bot added the a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) label Jan 8, 2025

chunhtai force-pushed the issues/158399-r3 branch from 80990ed to 1b9b5f1 Compare January 9, 2025 19:01

cbracken reviewed Jan 10, 2025

View reviewed changes

engine/src/flutter/runtime/runtime_controller.h Outdated Show resolved Hide resolved

cbracken reviewed Jan 10, 2025

View reviewed changes

chunhtai force-pushed the issues/158399-r3 branch from 4123a4c to fb23357 Compare January 13, 2025 19:07

chunhtai requested a review from cbracken January 13, 2025 19:10

chunhtai force-pushed the issues/158399-r3 branch from e4daa91 to ffd43a5 Compare January 31, 2025 23:54

goderbauer reviewed Feb 3, 2025

View reviewed changes

chunhtai force-pushed the issues/158399-r3 branch 2 times, most recently from 0346c47 to 4428901 Compare February 14, 2025 20:42

chunhtai force-pushed the issues/158399-r3 branch 2 times, most recently from b3d00a8 to 0fee10f Compare March 17, 2025 21:48

cbracken reviewed Mar 17, 2025

View reviewed changes

chunhtai force-pushed the issues/158399-r3 branch from 0fee10f to 8ca0563 Compare March 18, 2025 20:52

chunhtai requested a review from cbracken March 18, 2025 20:56

cbracken approved these changes Mar 18, 2025

View reviewed changes

chunhtai added the autosubmit Merge PR when tree becomes green via auto submit App label Mar 18, 2025

auto-submit bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Mar 18, 2025

chunhtai added the autosubmit Merge PR when tree becomes green via auto submit App label Mar 18, 2025

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 28, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

22b6daa

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 28, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

d7ceda3

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 28, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

1898212

…er/flutter#161265)

Piinks mentioned this pull request Mar 28, 2025

Flutter Notable Commits - 2025 #161460

Open

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 29, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

7a3ae43

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 29, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

c24b16c

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 30, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

6cae280

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 30, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

08f0978

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

67bada1

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

3dd1f40

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

c4af670

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

3acbf93

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

2cefb2e

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

f551f98

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

6011beb

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

84cae54

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

e29619b

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

0fb86b0

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

82088b5

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 20, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

867da66

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 20, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

42d9de7

…er/flutter#161265)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 21, 2025

26037df Add set semantics enabled API and wire iOS a11y bridge (flutt…

1b39da5

…er/flutter#161265)

chunhtai added a commit to chunhtai/flutter that referenced this pull request Jun 25, 2025

Reapply "Add set semantics enabled API and wire iOS a11y bridge (flut…

ff0d45b

…ter#161265)" This reverts commit cc04ca4.

chunhtai added a commit to chunhtai/flutter that referenced this pull request Jun 27, 2025

Reapply "Add set semantics enabled API and wire iOS a11y bridge (flut…

512ddf2

…ter#161265)" This reverts commit cc04ca4.

mdebbar mentioned this pull request Jul 17, 2025

Reapply "Add set semantics enabled API and wire iOS a11y bridge (#161… #171198

Merged

9 tasks

chunhtai added a commit to chunhtai/flutter that referenced this pull request Aug 1, 2025

Reapply "Add set semantics enabled API and wire iOS a11y bridge (flut…

10722ec

…ter#161265)" This reverts commit cc04ca4.

chunhtai added a commit to chunhtai/flutter that referenced this pull request Aug 14, 2025

Reapply "Add set semantics enabled API and wire iOS a11y bridge (flut…

e81a037

…ter#161265)" This reverts commit cc04ca4.

chunhtai added a commit to chunhtai/flutter that referenced this pull request Aug 18, 2025

Reapply "Add set semantics enabled API and wire iOS a11y bridge (flut…

f58cb87

…ter#161265)" This reverts commit cc04ca4.

	/// method to notify the engine such event.
	/// method to notify the engine whether the framework will generate a semantics tree.

Add set semantics enabled API and wire iOS a11y bridge #161265

Add set semantics enabled API and wire iOS a11y bridge #161265

Uh oh!

Conversation

chunhtai commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

Uh oh!

cbracken left a comment

Choose a reason for hiding this comment

Uh oh!

chunhtai commented Jan 13, 2025

Uh oh!

goderbauer left a comment

Choose a reason for hiding this comment

Uh oh!

goderbauer Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

justinmc commented Feb 26, 2025

Uh oh!

cbracken left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbracken left a comment

Choose a reason for hiding this comment

Uh oh!

auto-submit bot commented Mar 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chunhtai commented Jan 7, 2025 •

edited

Loading