Skip to content

Commit 0ea8af4

Browse files
authored
Split DevTools customer tests into its own customer_testing_devtools shard (flutter#195871)
`Linux customer_testing` runs the whole https://github.com/flutter/tests registry serially and is the longest framework presubmit at ~36 minutes. The two DevTools entries build DevTools from source and account for 15.5 of those minutes. Everything else is 20. This PR updates customer_testing to have `--include` and `--exclude` globs over the registry file names. `customer_testing` excludes `flutter_devtools*` and a new `Linux customer_testing_devtools` shard runs only those. The DevTools test step is Linux-only since the tests [currently only run on Linux](https://github.com/flutter/tests/blob/main/registry/flutter_devtools.test#L22-L24). The registry entries cloned `flutter/devtools` on every host before skipping, so Mac and Windows drop about 3 minutes too. Expected presubmit wall time for the shard: Linux 36 → ~23 min, with customer_testing_devtools at ~19. Mac 17 → ~14, Windows 17 → ~14. Will need a follow-up to take it out of bringup. `Linux customer_testing` on this PR took 21 minutes: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20customer_testing/134304/ Fixes flutter#164140 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 3cc7361 commit 0ea8af4

8 files changed

Lines changed: 98 additions & 12 deletions

File tree

‎.ci.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -594,6 +594,21 @@ targets:
594594
# tests are sharded.
595595
test_timeout_secs: "3600" # Allows 60 minutes (up from 30 default)
596596

597+
# The DevTools tests build DevTools from source and take as long as the rest
598+
# of the registry, and their test step only runs on Linux.
599+
- name: Linux customer_testing_devtools
600+
bringup: true
601+
enabled_branches:
602+
- master
603+
recipe: flutter/flutter_drone
604+
# Timeout in minutes for the whole task.
605+
timeout: 60
606+
properties:
607+
shard: customer_testing_devtools
608+
tags: >
609+
["framework", "hostonly", "shard", "linux"]
610+
test_timeout_secs: "2700" # Allows 45 minutes (up from 30 default)
611+
597612
# Despite the name, this really means "generate api_docs.zip", and "upload zip
598613
# to GCS", and conditionally on the "master" channel will also deploy the docs
599614
# to Firebase (https://main-api.flutter.dev/).

‎TESTOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,7 @@
372372
# ci_yaml flutter roller @keyonghan @flutter/infra
373373
# coverage @christopherfujino @flutter/infra
374374
# customer_testing @justinmc @flutter/framework
375+
# customer_testing_devtools @justinmc @flutter/framework
375376
# docs @justinmc @flutter/framework
376377
# snippets @justinmc @flutter/framework
377378
# flutter_packaging @christopherfujino @flutter/infra

‎dev/bots/suite_runners/run_customer_testing_tests.dart‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,17 @@ import 'package:path/path.dart' as path;
99
import '../run_command.dart';
1010
import '../utils.dart';
1111

12-
Future<void> customerTestingRunner() async {
12+
// The DevTools tests build DevTools from source and take longer than the rest
13+
// of the registry combined, so they get their own shard.
14+
const String _devtoolsTestsGlob = 'flutter_devtools*.test';
15+
16+
Future<void> customerTestingRunner() =>
17+
_runCustomerTesting(<String>['--exclude=$_devtoolsTestsGlob']);
18+
19+
Future<void> customerTestingDevtoolsRunner() =>
20+
_runCustomerTesting(<String>['--include=$_devtoolsTestsGlob']);
21+
22+
Future<void> _runCustomerTesting(List<String> ciArguments) async {
1323
printProgress('${green}Running customer testing$reset');
1424
await runCommand('git', const <String>[
1525
'fetch',
@@ -29,7 +39,7 @@ Future<void> customerTestingRunner() async {
2939
}
3040
await runCommand(
3141
path.join(flutterRoot, 'dev', 'customer_testing', Platform.isWindows ? 'ci.bat' : 'ci.sh'),
32-
<String>[],
42+
ciArguments,
3343
workingDirectory: path.join(flutterRoot, 'dev', 'customer_testing'),
3444
);
3545
}

‎dev/bots/test.dart‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ Future<void> main(List<String> args) async {
155155
'flutter_plugins': flutterPackagesRunner,
156156
'skp_generator': skpGeneratorTestsRunner,
157157
'customer_testing': customerTestingRunner,
158+
'customer_testing_devtools': customerTestingDevtoolsRunner,
158159
'analyze': analyzeRunner,
159160
'fuchsia_precache': fuchsiaPrecacheRunner,
160161
'snippets': _runSnippetsTests,

‎dev/customer_testing/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,12 @@ cd dev/customer_testing
1212
dart ci.dart [sha]
1313
```
1414

15+
To run a subset of the registry, pass `--include=<glob>` or `--exclude=<glob>`,
16+
matched against the `registry/*.test` file names:
17+
18+
```sh
19+
dart ci.dart --include=flutter_devtools*.test
20+
```
21+
1522
Or, to update the SHA for our CI, edit and send a PR for
1623
[`tests.version`](tests.version).

‎dev/customer_testing/ci.bat‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,4 @@ ECHO Updating pub packages...
2020
CALL dart pub get
2121

2222
REM Run the cross-platform script.
23-
CALL ..\..\bin\dart.bat ci.dart
23+
CALL ..\..\bin\dart.bat ci.dart %*

‎dev/customer_testing/ci.dart‎

Lines changed: 60 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,32 +4,61 @@
44

55
import 'dart:io' as io;
66

7+
import 'package:args/args.dart';
8+
import 'package:glob/glob.dart';
79
import 'package:path/path.dart' as p;
810

911
/// To run this script, either:
1012
///
1113
/// ```sh
1214
/// cd dev/customer_testing
13-
/// dart ci.dart [sha]
15+
/// dart ci.dart [--include=<glob>]... [--exclude=<glob>]... [sha]
1416
/// ```
1517
///
1618
/// Or:
1719
///
1820
/// ```sh
19-
/// ./dev/customer_testing/ci.sh
20-
/// ./dev/customer_testing/ci.bat
21+
/// ./dev/customer_testing/ci.sh [args...]
22+
/// ./dev/customer_testing/ci.bat [args...]
2123
/// ```
24+
///
25+
/// `--include` and `--exclude` are globs matched against `registry/*.test`
26+
/// file names, used to split the registry across CI shards.
2227
void main(List<String> args) async {
28+
final argParser = ArgParser()
29+
..addMultiOption(
30+
'include',
31+
help: 'Only run registry files whose name matches this glob. Can be repeated.',
32+
valueHelp: 'glob',
33+
)
34+
..addMultiOption(
35+
'exclude',
36+
help: 'Skip registry files whose name matches this glob. Can be repeated.',
37+
valueHelp: 'glob',
38+
);
39+
40+
final ArgResults parsedArgs;
41+
try {
42+
parsedArgs = argParser.parse(args);
43+
} on ArgParserException catch (error) {
44+
io.stderr.writeln(error.message);
45+
io.stderr.writeln('Usage: dart ci.dart [--include=<glob>]... [--exclude=<glob>]... [sha]');
46+
io.exitCode = 1;
47+
return;
48+
}
49+
2350
final String sha;
24-
if (args.isEmpty) {
51+
if (parsedArgs.rest.isEmpty) {
2552
sha = io.File('tests.version').readAsStringSync().trim();
26-
} else if (args.length == 1) {
27-
sha = args.first;
53+
} else if (parsedArgs.rest.length == 1) {
54+
sha = parsedArgs.rest.first;
2855
} else {
29-
io.stderr.writeln('Usage: dart ci.dart [sha]');
56+
io.stderr.writeln('Usage: dart ci.dart [--include=<glob>]... [--exclude=<glob>]... [sha]');
3057
io.exitCode = 1;
3158
return;
3259
}
60+
final List<Glob> includes = parsedArgs.multiOption('include').map(Glob.new).toList();
61+
final List<Glob> excludes = parsedArgs.multiOption('exclude').map(Glob.new).toList();
3362

3463
final String flutterRootPath = p.canonicalize('../../');
3564
final testsCacheDir = io.Directory(p.join(flutterRootPath, 'bin', 'cache', 'pkg', 'tests'));
@@ -74,13 +103,36 @@ void main(List<String> args) async {
74103
return;
75104
}
76105

106+
final registryDir = io.Directory(p.join(testsCacheDir.path, 'registry'));
107+
// run_tests.dart parses its arguments as globs, so avoid backslashes.
108+
final List<String> testFiles =
109+
registryDir
110+
.listSync()
111+
.whereType<io.File>()
112+
.map((io.File file) => p.basename(file.path))
113+
.where((String name) {
114+
return p.extension(name) == '.test' &&
115+
(includes.isEmpty || includes.any((Glob glob) => glob.matches(name))) &&
116+
!excludes.any((Glob glob) => glob.matches(name));
117+
})
118+
.map((String name) => p.posix.joinAll(<String>[...p.split(registryDir.path), name]))
119+
.toList()
120+
..sort();
121+
if (testFiles.isEmpty) {
122+
io.stderr.writeln(
123+
'No registry files matched --include=$includes --exclude=$excludes in ${registryDir.path}',
124+
);
125+
io.exitCode = 1;
126+
return;
127+
}
128+
77129
io.stderr.writeln('Running tests...');
78130
final io.Process test = await io.Process.start('dart', <String>[
79131
'--enable-asserts',
80132
'run_tests.dart',
81133
'--skip-on-fetch-failure',
82134
'--skip-template',
83-
p.posix.joinAll(<String>[...p.split(testsCacheDir.path), 'registry', '*.test']),
135+
...testFiles,
84136
], mode: io.ProcessStartMode.inheritStdio);
85137
if ((await test.exitCode) != 0) {
86138
io.exitCode = 1;

‎dev/customer_testing/ci.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,4 +32,4 @@ cd "$(script_location)"
3232
dart pub get
3333

3434
# Run the cross-platform script.
35-
../../bin/dart run ci.dart
35+
../../bin/dart run ci.dart "$@"

0 commit comments

Comments
 (0)