Skip to content

[chore] Split up long-running tool_integration_tests to have more balanced sharding #185013

Closed

Description

@dcharkes

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8684718470335776881/+/u/run_test.dart_for_tool_integration_tests_shard_and_subshard_7_10/stdout
This shard runs 60 minutes, 45 minutes in the tests.

https://ci.chromium.org/ui/p/flutter/builders/try/Windows%20tool_integration_tests_8_10/132/overview
This is the fasted shard. 3.5 minutes for the actual tests - and 9 minutes for the total bot. So there's 5.5 minutes of overhead for setting up the bot before running tests.

I think just creating more shards or giving a longer timeout to the shards is the wrong solution here. Instead we should try to get the tests more balanced across the shards.

Some ideas for solving this issue:

  1. Change tests that run multiple tests in one file to separate test files, so they get distributed more.
  2. Create a list of long-running tests in _runIntegrationToolTests and forcibly assign them to different shards.

Yeah, I think 1. is the best solution. Realistically, we shouldn't be placing multiple integration tests in the same file, but we haven't really been consistent with that

By @bkonyi

Activity

  1. saurabh-mirajkar commented on Apr 14, 2026

    @saurabh-mirajkar
    Contributor

    @dcharkes Thanks for the report. Labelling for team-infra to investigate further.

  2. added
    a: tests"flutter test", flutter_test, or one of our tests
    team-infraOwned by Infrastructure team
    f: integration_testThe flutter/packages/integration_test plugin
    and removed
    in triagePresently being triaged by the triage team
    on Apr 14, 2026
  3. jtmcdole commented on Apr 20, 2026

    @jtmcdole
    Member

    team-infra is not the right team.

  4. added
    team-toolOwned by Flutter Tool team
    and removed
    team-infraOwned by Infrastructure team
    on Apr 20, 2026
  5. dcharkes commented on Apr 21, 2026

    @dcharkes
    ContributorAuthor
    Image

    11 to 41 minutes. Splitting up the tests I own at least prevents timeouts now.

    @bkonyi I think this is for you to decide whether you want to prio tracking down other tests that could use splitting up from the 41 and 37 minute bots.

  6. bkonyi commented on Apr 21, 2026

    @bkonyi
    Contributor

    Yeah, I think this is enough evidence to show that smaller test libraries results in increased granularity when it comes to sharding, which should better distribute slower tests across all the shards. We should really get into the habit as a team to create multiple test files for features rather than shoving multiple related tests into the same library.

    This is something I can take on. I'll start with the integration tests, but will also investigate how we can restructure the tool's test suite overall.

  7. self-assigned this
    on Apr 21, 2026
  8. added
    P2Important issues not at the top of the work list
    triaged-toolTriaged by Flutter Tool team
    on Apr 21, 2026
  9. added
    ⏳Bot is counting down the days until it unassigns the issue
    on Aug 25, 2026
  10. flutter-triage-bot commented on Aug 25, 2026

    @flutter-triage-bot

    This issue is assigned to @bkonyi but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

  11. jmagman commented on Oct 29, 2026

    @jmagman
    Member

    Hey @bkonyi this is assigned to you but I had an idea for how to do this: #195628

  12. removed
    ⏳Bot is counting down the days until it unassigns the issue
    on Oct 29, 2026
  13. added a commit that references this issue on Nov 2, 2026
    3f19e89
  14. added a commit that references this issue on Nov 3, 2026
    0df68ad
  15. github-actions commented on Nov 16, 2026

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  16. locked as resolved and limited conversation to collaborators on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Important issues not at the top of the work lista: tests"flutter test", flutter_test, or one of our testsf: integration_testThe flutter/packages/integration_test pluginteam-toolOwned by Flutter Tool teamtriaged-toolTriaged by Flutter Tool team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions