Skip to content

[tool][material_ui] Only run test files containing the package's tag with dart-test --package-tags (currently just material_ui web tests) - #13536

Open
elliette wants to merge 5 commits into
flutter:mainfrom
elliette:flutter-js-tests
Open

elliette wants to merge 5 commits into
flutter:mainfrom
elliette:flutter-js-tests

Conversation

@elliette

@elliette elliette commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Workaround for flutter/flutter#195700
Part of flutter/flutter#192308


Note: This PR reverts #13537, re-enabling the material_ui web unit tests.


When a package has an entry in --package-tags (currently just material_ui, for the web unit tests), dart-test now passes flutter test (or dart test) only the *_test.dart files that contain the tag, instead of the whole test/ directory. Before, every test file was compiled and loaded just to find out whether it had any tagged tests. Now only the files that mention the tag are compiled and loaded.

This matters most on master, where flutter/flutter#193485 roughly doubled how long each web test suite takes to load. That pushed the shard running the material_ui web tests past its 90-minute timeout.

With this change, material_ui loads 52 test files on web instead of 540, and its web tests are about 13× faster.

Before (#26883) After (#26894, this PR)
Test files loaded (material_ui, example, test_apps/a11y_assessments) 540 52
Tests run 176 + 4 176 + 4
material_ui web test time, master ~74 min ~6 min
Whole shard, master 82 min 14 min
material_ui web test time, stable (#26867 → #26871) ~22 min ~3 min

Caveats

1. The tag must appear literally in the test file

Test files are chosen by searching their text for the tag name. So the tag must be written out in each test file that uses it.

This works
// in my_web_test.dart
testWidgets('My web test', (WidgetTester tester) {
   expect(somethingWebby(), isTrue);
 }, tags: 'reduced-web-test-set');
This does not
// in shared_web_utils.dart
const webTestTag = 'reduced-web-test-set';

// in my_web_test.dart
testWidgets('My web test', (WidgetTester tester) {
  expect(somethingWebby(), isTrue);
}, tags: webTestTag);

To catch this, dart-test now fails a tagged package if the tag appears in any non-test Dart file in its lib/ or test/ directories, and lists the offending files.

2. Falls back to old behavior (load/compile all test files) if it's not safe to only load/compile the tagged files

In certain cases, a simple string search isn't sufficient to find all tagged tests. In that case, we fallback to loading and compiling all test files. This happens when:

  • the tag isn't a single tag name (e.g. !slow, a || b);
  • dart_test.yaml contains any of add_tags, paths, filename or include keys.

Finally, a tagged package with no test files containing the tag is now skipped with "No test files contain the requested tag." instead of starting flutter test. For example, material_ui/test_apps/a11y_assessments used to load 45 suites and run 0 tests.

Pre-Review Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

Footnotes

  1. Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. ↩ ↩2

@elliette elliette added the CICD Run CI/CD label Nov 2, 2026
@github-actions github-actions Bot added p: material_ui triage-design Should be looked at in design triage labels Nov 2, 2026
Reverts flutter#13537, which temporarily excluded material_ui from the non-Wasm
web unit tests because the shard containing it was exceeding its 90 minute
timeout. With dart-test --package-tags now only loading the test files that
contain the package's tag, material_ui's web tests take ~6 minutes instead
of ~74 on master.
@elliette
elliette marked this pull request as ready for review November 3, 2026 18:34
@elliette elliette changed the title [tool] Only run test files containing the package's tag with dart-test --package-tags [tool][material_ui] Only run test files containing the package's tag with dart-test --package-tags (currently just material_ui web tests) Nov 3, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the dart-test command in the repository tooling to optimize running tests with package tags. Instead of loading and compiling all test files to filter by tag, the tool now scans and runs only the specific test files that literally contain the tag name, falling back to the full test suite filter if tag selector expressions or certain dart_test.yaml configurations are used. Feedback on the changes suggests checking if the test directory exists before listing its contents in _testFilesForTag to prevent a potential FileSystemException crash on packages without a test directory.

@QuncCccccc
QuncCccccc self-requested a review November 3, 2026 22:22

@QuncCccccc QuncCccccc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the improvement!

@mdebbar mdebbar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing wrong with the PR, I just want to try a fix upstream before we commit to a workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD p: material_ui triage-design Should be looked at in design triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants