To guard against mistakes where packages have tests that are misconfigured (e.g., in the wrong directory) or accidentally missing, many types of tests in flutter/packages are treated as CI errors if they are missing from any package, and a specific exclusion list is used to override that. This system is also used to allow temporarily disabling a whole type of test for a package due (e.g., to unblock the tree when a roll regression breaks a specific kind of test for one package, and can't be easily reverted in flutter/flutter).

Currently the way the exclusions work is that we have a set of exclusion config files in script/config, and the CI invocations pass the relevant exclusion file. So for instance, the CI step that runs web integration tests passes exclude_integration_web.yaml. This approach has some benefits:

• We can easily see all the mass-disabled tests in one place.
• We can add specialized exclusions as easily as we add specialized CI steps. For instance, when a variant of the web integration tests was set up that uses the --wasm flag, we just added exclude_integration_web_wasm.yaml, and passed both exclusion lists to the new setup.
• Running locally bypasses the exclusions, making it easy to run tests locally, for debugging, that are disabled in CI.

It has a couple of drawbacks, however:

• Running locally bypasses the exclusions, which can be surprising if you don't realize that the reason a test fails locally is that it was already failing, it's just not run in CI.
• The system for excluding expensive runs when files are touched that don't affect them treats these as CI files and triggers a full re-run (and changing that would require adding a lot more complicated logic that looked at the actual diffs).

An alternative would be a package-level configuration file read by repo-tools that configured these exceptions locally to the package. For instance, we could replace exclude_integration_web.yaml with a file called something like repo_tools_config.yaml in each package currently on that list, with something like:

exclude:
  integration_test:
    - web

This would allow packages to contain all of their configs in one place, so you could easily see all the tests a given package opts out of, and adding or removing an exclusion wouldn't trigger a full-repo test run. The downside is that the config would be more complex, and we would need to build support for each specific kind of exclusion we wanted; adding the wasm-specific exclusion, for instance, would have required new syntax, and tool support for it.

That's probably a worthwhile tradeoff to make, since adding/removing individual exclusions is much more common than creating new types of exclusions.