Repository navigation
[packages] Test infra: Skipping all tests for a package is considered a failure #191495
Description
Context: I'm trying to exclude the material_ui/tool/gen_defaults tests from running on web, because gen_defaults is a CLI utility.
I've trying using a @TestOn annotation and a dart_test.yaml file to exclude them. In both cases, although the tests are correctly skipped on web, they get reported as a failure.
- Adding
@TestOn(!'chrome')annotation: flutter/packages@10b8d80 - Adding a
dart_test.yamlfile: flutter/packages@f082ed1
See logs:
Expected behavior: if all tests for a package are skipped for a platform, it is not a failure
Actual behavior: if all tests for a package are skipped for a platform, it is a failure
Related: #191447
Activity
stuartmorgan-g commentedon Aug 24, 2026
Hm, this should have worked. We've run into this exact scenario before, where we want to skip all tests just on web, but
dart/flutter testitself returns a failure code when all tests are skipped. At the time at least I didn't see a way to disable that behavior, or to robustly detect it (i.e., a unique and documented return code that we could ignore). Instead I implemented a rudimentary parser for dart_test.yaml to identify tests that are intended to be web-only or non-web only, and then skip trying to run them.You could re-test locally with some logging added to that part of the tooling and see why it didn't work. Maybe we need to actually parse the yaml file as yaml, and implement better matching on the values?
stuartmorgan-g commentedon Aug 24, 2026
Ugh, I see the problem. You used quotes on the value, and I didn't account for that.
This is a classic case of:
If I'd just YAML-parsed this in the first place, this would have worked when you tried it. I'll fix the tool.
[tool] Fix dart_test.yaml parsing (#12574)
[tool] Fix dart_test.yaml parsing (flutter#12574)
github-actions commentedon Sep 7, 2026
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.