Add extension points to the test runner to support user defined behaviour #2225

Coming from a Python background where I have made extensive use of the pytest library, I became accustomed to the level of customizability the library offered. One of the most powerful / useful features of it are it's extensible fixtures. These allow users to heavily customize the mechanics of the test runner via a set of hooks / extension points, allowing custom behaviour to be "injected" into various parts of the runner sequencing. Some of the most notable features are:

• being able to override test fixtures on a folder basis
• being able to extend command line parameters for the test runner to customize fixture behaviour
• being able to scope test fixtures so you can customize behaviour for specific tests, classes, or modules ... or an entire suite.
• hooking into the test runner collection phase so you can customize the test behaviour depending on which tests are being run in the current invocation

These are just a few examples of the kinds of extension points that I haven't been able to find comparables for in the Dart testing framework. I did notice that there were some plans to add support for plugins to the test runner, but I'm not sure the tasks I found there offer nearly the same level of customization that other test frameworks like PyTest offer. In particular, the ones that allow for scoping of test fixtures to different levels, and hooking into specific stages with the test orchestration pipeline. Such features greatly enhance the ability of developers to customize the behaviour for unique workflows.

Below are some workflows that I've used personally in other test suites that would benefit greatly from having this flexibility:

• being able to aggregate log output either per-test, per-module or per suite/subset of tests
• being able to orchestrate shared state that is only required within certain scopes (ie: shared credentials for a group of tests, shared resources like Docker environments, etc.)
• being able to optimize the use of expensive resources depending on test state (ie: consider a test suite with 100 tests, 90 of which are light weight tests and use mocks, and 10 of which use heavy resources like docker container. Being able to reserve the creation / destruction of the expensive resources ONLY when tests that require them ARE SELECTED for execution in the current run)

I know this enhancement request is intentionally open ended, but I thought it might be good to create a starting point for discussion on various options for adding extension points like what I've just described and then maybe branch it off to more specific line items depending on which ones are most feasible.