Skip to content

Split DevTools customer tests into its own customer_testing_devtools shard - #195871

Merged
auto-submit[bot] merged 2 commits into
flutter:masterfrom
jmagman:customer-testing-devtools-shard
Nov 9, 2026
Merged

auto-submit[bot] merged 2 commits into
flutter:masterfrom
jmagman:customer-testing-devtools-shard

Conversation

@jmagman

@jmagman jmagman commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Linux customer_testing runs the whole https://github.com/flutter/tests registry serially and is the longest framework presubmit at ~36 minutes. The two DevTools entries build DevTools from source and account for 15.5 of those minutes. Everything else is 20.

This PR updates customer_testing to have --include and --exclude globs over the registry file names. customer_testing excludes flutter_devtools* and a new Linux customer_testing_devtools shard runs only those.

The DevTools test step is Linux-only since the tests currently only run on Linux. The registry entries cloned flutter/devtools on every host before skipping, so Mac and Windows drop about 3 minutes too.

Expected presubmit wall time for the shard: Linux 36 → ~23 min, with customer_testing_devtools at ~19. Mac 17 → ~14, Windows 17 → ~14. Will need a follow-up to take it out of bringup.

Linux customer_testing on this PR took 21 minutes: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20customer_testing/134304/

Fixes #164140

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

@jmagman jmagman added a: devtools DevTools related - suite of performance and debugging tools CICD Run CI/CD labels Nov 3, 2026
@jmagman

jmagman commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces sharding for customer testing by splitting out DevTools tests into their own CI target (customer_testing_devtools). It updates the CI configuration, test owners, and test runners to support --include and --exclude glob arguments. Feedback on the changes highlights a potential path-handling bug on Windows and a missing directory check in ci.dart, as well as a style guide violation regarding missing public documentation on the new runner functions.

Comment thread dev/customer_testing/ci.dart Resolved
@jmagman
jmagman marked this pull request as ready for review November 3, 2026 07:07

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces sharding for customer testing by adding support for --include and --exclude glob filters in the ci.dart script, allowing the DevTools tests to run in a separate CI shard. Feedback suggests simplifying the path normalization logic in ci.dart to be more robust on Windows by replacing backslashes directly instead of splitting and joining paths.

Comment thread dev/customer_testing/ci.dart Resolved
@jmagman
jmagman requested a review from kenzieschmoll November 3, 2026 19:27
@jmagman jmagman added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Nov 3, 2026
Comment thread dev/customer_testing/ci.dart Resolved
Comment thread dev/customer_testing/ci.dart Resolved
kenzieschmoll
kenzieschmoll previously approved these changes Nov 3, 2026

@kenzieschmoll kenzieschmoll left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with a couple of nits

@github-actions github-actions Bot removed the c: contributor-productivity Team-specific productivity, code health, technical debt. label Nov 4, 2026
@jmagman jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 9, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Nov 9, 2026
Merged via the queue into flutter:master with commit 0ea8af4 Nov 9, 2026
21 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Nov 9, 2026
jmagman added a commit to jmagman/flutter that referenced this pull request Nov 10, 2026
These overrides date from when the shards ran 30-45 minutes. After
flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so
the 30 minute default is enough and a hang fails sooner.
auto-submit Bot pushed a commit to flutter/packages that referenced this pull request Nov 10, 2026
flutter/flutter@d463821...fffd8c2

2026-11-10 [email protected] Roll Fuchsia Test Scripts from 8QcsP_SmbsT09VgUs... to PXc5D7S_q6sIYpfFJ... (flutter/flutter#196273)
2026-11-10 [email protected] [web] Accept the capped letter spacing line shift in kitchensink (flutter/flutter#196148)
2026-11-10 [email protected] Fix Vulkan MSAA diagonal seam in FramebufferBlend (flutter/flutter#196197)
2026-11-10 [email protected] Read back images on the raster thread on OpenGLES (flutter/flutter#195927)
2026-11-10 [email protected] [web] Unify ui.Paint across CanvasKit and Skwasm (flutter/flutter#196071)
2026-11-10 [email protected] Roll Fuchsia Test Scripts from p3N3B-Kchh1bthB6n... to 8QcsP_SmbsT09VgUs... (flutter/flutter#196249)
2026-11-09 [email protected] [Impeller] Reland clipped BackdropFilter square corners fix (flutter/flutter#195716)
2026-11-09 [email protected] [web] Recreate canvas and context on CkOffscreenSurface resize to fix WebGL clipping (flutter/flutter#192769)
2026-11-09 [email protected] Roll Dart SDK from ee0f66fa0ad2 to a2baae273bb2 (1 revision) (flutter/flutter#196245)
2026-11-09 [email protected] Add Mac web_tool_tests_1_2 and _2_2 in bringup (flutter/flutter#196241)
2026-11-09 [email protected] Roll pub packages (flutter/flutter#196246)
2026-11-09 [email protected] Increase number of 'Windows tool_integration_tests' subshards (flutter/flutter#196243)
2026-11-09 [email protected] Roll Fuchsia Test Scripts from Jo-ZgQSNDuPsjlZ4s... to p3N3B-Kchh1bthB6n... (flutter/flutter#196238)
2026-11-09 [email protected] Skip UIScene pre-migration test cases on Xcode 27 (flutter/flutter#194755)
2026-11-09 [email protected] [flutter_tools] Share Analytics.isBot and make isRunningOnBot sync (flutter/flutter#194744)
2026-11-09 [email protected] [flutter_tools] Retry chromedriver startup when its port is taken (flutter/flutter#196150)
2026-11-09 [email protected] Remove unused vulkan_provider.cc (flutter/flutter#195513)
2026-11-09 [email protected] Roll Dart SDK from 3b174b7217fe to ee0f66fa0ad2 (13 revisions) (flutter/flutter#196217)
2026-11-09 [email protected] Goma is dead, long live RBE! (Take 2) (flutter/flutter#196212)
2026-11-09 [email protected] Split DevTools customer tests into its own customer_testing_devtools shard (flutter/flutter#195871)
2026-11-09 [email protected] Split mac_host_{debug,profile,release}_arm64_tests into one step per type (flutter/flutter#196031)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
robert-ancell pushed a commit to robert-ancell/flutter that referenced this pull request Nov 10, 2026
Introduced in flutter#195871. Is passing
in post-submit:

https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20customer_testing_devtools/1/overview
(17 minutes)

Also drop `test_timeout_secs` which was copied from `customer_testing`.
The 30 minute default is now enough.


## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
@jmagman
jmagman deleted the customer-testing-devtools-shard branch November 11, 2026 00:03
@jmagman jmagman mentioned this pull request Nov 11, 2026
10 of 11 tasks
kthguru pushed a commit to infradise/flutter that referenced this pull request Nov 11, 2026
…6277)

These overrides date from when the shards ran 30-45 minutes. After
flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so the
30 minute default is enough and a hang fails sooner.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
@jmagman jmagman added the cp: beta cherry pick this pull request to beta release candidate branch label Nov 11, 2026
hariuc pushed a commit to hariuc/flutter that referenced this pull request Nov 12, 2026
Flame only runs its full test suites on Linux, where it takes 7–9 min,
as long as the rest of the registry combined. Give it its own Linux
builder, the same way `customer_testing_devtools` was split out in
flutter#195871. Mac and Windows are
unchanged since flame is analysis-only there.

`Linux customer_testing` is the last build to finish in about half of
framework presubmit runs. It goes from ~22 min to ~13, and `Linux
customer_testing_flame` runs in ~10.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: devtools DevTools related - suite of performance and debugging tools CICD Run CI/CD cp: beta cherry pick this pull request to beta release candidate branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tests] Run DevTools customer tests in their own shard

2 participants