feedback on the new preprocessSuite API #41708

Hi folks,

Thank you for the new custom reporter planning API!

We tried it out as part of the our shard balancing algorithm in the https://github.com/flakiness/playwright. Our balancer relies on the previous test durations to allocate well-balanced shards.

With the new API surface, we can't properly measure the weight of the setup / teardown projects since we're not given the test cases in these projects:

• fullConfig exposes the project dependency tree, but not the test cases inside these projects that will be executed
• the suite exposes only top-level project suites and their test cases, not the test cases inside setup/teardown projects

As a result, a commit that drops / adds some tests in the setup or teardown projects will not result in shard re-balancing.

Any chance the API can be enhanced to expose more information to clients?

Thanks,
Andrey

P.S. As of today, we're able to get tests from the setup/teardown projects: we have the flakiness-playwright-shard binary that runs playwright with --list to get all the test cases scheduled to run, then fetches durations for them, and then allocates shards. It'd be nice to match this behavior to get rid of the multiple hops we currently do.