Add globalSetup and addGlobalTearDown APIs for cross-suite setup and teardown by sigurdm · Pull Request #2714 · dart-lang/test

sigurdm · 2026-08-05T15:06:03Z

Adds globalSetup() and addGlobalTearDown() APIs in package:test_api / package:test to execute shared global setup scripts once on the host test runner, memoizing results across all test suites and executing registered teardowns when the test runner closes.

Addresses #2225

The main motivation is to support setup shared across multiple test suites (_test.dart files). Examples include:

Precompiling a binary used by multiple end-to-end tests
Provisioning test databases or starting local test servers

Example

// test/setup_server.dart
import 'dart:io';
import 'package:test/global.dart';

Future<Map<String, dynamic>> setUp() async {
  var server = await HttpServer.bind('localhost', 0);
  addGlobalTearDown(() async => await server.close(force: true));
  return {'port': server.port};
}

// test/service_test.dart
import 'package:test/test.dart';

void main() {
  test('connects to server', () async {
    final config = await globalSetup(
      Uri(path: '/test/setup_server.dart'),
    ) as Map;
    final port = config['port'] as int;
    // ...
  });
}

Key Features

globalSetup(Uri uri):
- Runs the Dart file at uri in an isolate on the host runner and returns its JSON-encodable result.
- Memoizes the JSON-encoded result so the script only executes once per canonical URI, while returning an independent decoded value to each caller.
- Must be called from within a running test or a setUp() / setUpAll() callback.
- Bootstraps isolates via an in-memory data: URI using Isolate.spawnUri (sharing architecture with spawnHybridUri), avoiding filesystem compilation or runner-specific compiler hooks.
addGlobalTearDown(FutureOr<void> Function() callback) (exported from package:test/global.dart):
- Called from within a setup script to register cleanup tasks.
- Executed in reverse registration order (LIFO) when the test runner closes.
- Best-effort cleanup: all registered teardowns run even if an earlier teardown throws, with errors reported to stderr and runner exiting with code 1.
- 30-second per-isolate timeout to prevent hanging teardowns.
URI Resolution:
- Root-relative (Uri(path: '/test/setup.dart')): Resolved relative to package root (pubspec.yaml).
- package: URIs (Uri.parse('package:my_pkg/test_helpers.dart')): Resolved synchronously via package config.
- Relative URIs (Uri(path: 'setup.dart') or Uri(path: '../setup.dart')): Resolved relative to the executing test suite's directory.
- file: URIs: Resolved as absolute filesystem paths.
- Fragments are automatically removed for consistent compilation caching. Relative, root-relative, and file URIs strictly prohibit query parameters.
Cross-Platform Support:
- Dispatches over test_channel, allowing browser, Node, and VM tests alike to trigger host-side setup.
Standalone Execution Support:
- Tests run directly via dart <test_file>.dart execute setup scripts using a standalone scaffolding fallback and run teardowns on process exit.
Flutter Compatibility:
- Works out of the box with flutter test without runner-level code changes.

github-actions · 2026-08-05T15:07:37Z

PR Health

Publish ✔️

Package	Version	Status
package:test ⭐	1.34.0-wip	WIP (no publish necessary)
package:test_api ⭐	0.7.16-wip	WIP (no publish necessary)
package:test_core ⭐	0.6.22-wip	WIP (no publish necessary)

3 already published.
3 WIP (no publish necessary).

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

This check can be disabled by tagging the PR with skip-publish-check.

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check (for example, for non-user-visible changes when the package version is already at -wip).

… fallback

…cleanup grace period

…ption catch blocks

… isolate-focused

…e.hook

…Down APIs

…sages

sigurdm · 2026-09-18T11:30:38Z

@natebosch are you ok with the PR as it stands?

…ager, and validate JSON return

…to script, and clean up exports

…r root-relative check

- Reflow long lines - Remove a src import - Remove an unnecessary type on the left

natebosch · 2026-10-08T22:40:55Z

This is still on my radar and I should be able to get back to land it soon - I wanted to get some of the other test changes published before landing it.

natebosch · 2026-11-05T23:51:41Z

+      if (mainFn is FutureOr<Object?> Function(List<String>, SendPort)) {
+        return await mainFn(args, sendPort);
+      } else if (mainFn is FutureOr<Object?> Function(List<String>)) {
+        return await mainFn(args);
+      } else if (mainFn is FutureOr<Object?> Function()) {
+        return await mainFn();
+      } else {
+        throw ArgumentError(
+          'The global setup script must define a top-level `setUp` function '
+          'with zero arguments, one List<String> argument, '
+          'or (List<String>, SendPort).',
+        );
+      }


The docs specify a zero-argument setUp() function, so let's only accept FutureOr<Object?> Function() here rather than supporting (List<String>, SendPort) and (List<String>). args is always [], sendPort exposes the runner's internal bootstrap response port, and once published we wouldn't be able to drop those extra signatures without risking a breaking change.

Done — restricted internalBootstrapVmHook to zero-argument FutureOr<Object?> Function() and removed the unused args parameter.

natebosch · 2026-11-05T23:51:44Z

+    try {
+      json.encode(result);
+    } on Object catch (error) {
+      throw ArgumentError(
+        'The global setup script returned a value that is not JSON-encodable: '
+        '$error',
+      );
+    }
+
+    sendPort.send({
+      'success': true,
+      'result': result,
+      'hasTearDown': tearDowns.isNotEmpty,
+      'commandPort': tearDowns.isNotEmpty ? commandPort.sendPort : null,
+    });


Right now json.encode(result) validates encodability here, then the raw result is sent over sendPort.send across the Isolate.spawnUri isolate-group boundary, and GlobalSetupManager / standalone_global_setup.dart call json.encode(result) a second time to cache the string.

If result is (or contains) an object with a toJson() method, json.encode(result) succeeds here and then sendPort.send fails because arbitrary class instances can't be sent across isolate groups. Sending the already-encoded JSON String in 'result' (so the JSON round-trip straddles the isolate boundary instead of happening on one side) fixes toJson() values and avoids encoding twice.

Good catch. internalBootstrapVmHook now sends the encoded JSON String in 'result', and GlobalSetupManager / standalone_global_setup.dart cache it directly without re-encoding. Added a test for objects with toJson().

natebosch · 2026-11-05T23:51:46Z

+  } catch (error, stack) {
+    commandPort.close();
+    sendPort.send({
+      'success': false,
+      'error': error.toString(),
+      'stackTrace': stack.toString(),
+    });
+  }


If setUp() starts a resource, registers an addGlobalTearDown callback, and then throws later in setUp() (or returns a value that fails json.encode), we enter catch, report 'success': false, and the runner kills the isolate without running the already-registered tearDowns. Should we run tearDowns.reversed on a best-effort basis before reporting failure so partially initialized resources get cleaned up?

Done — the catch block now runs tearDowns.reversed on a best-effort basis before closing commandPort and reporting failure. Added a test verifying cleanup on partial setup failure.

natebosch · 2026-11-05T23:52:09Z

+  Future<void> close() => _closeMemo.runOnce(() async {
+    for (var active in _activeSetups.reversed) {
+      final replyPort = RawReceivePort();
+      final completer = Completer<Map<Object?, Object?>>();
+      replyPort.handler = (Object? response) {
+        if (!completer.isCompleted) {
+          completer.complete(response as Map<Object?, Object?>);
+        }
+      };
+      try {
+        active.commandPort.send([replyPort.sendPort, 'teardown']);
+        final response = await completer.future.timeout(
+          const Duration(seconds: 30),
+          onTimeout: () => {'success': false, 'error': 'Teardown timed out'},
+        );
+        if (response['success'] != true) {
+          stderr.writeln(
+            'Global teardown for "${active.url}" failed: ${response["error"]}',
+          );
+          exitCode = 1;
+        }
+      } catch (error) {
+        stderr.writeln('Global teardown for "${active.url}" failed: $error');
+        exitCode = 1;
+      } finally {
+        replyPort.close();
+        active.isolate.kill();
+      }
+    }
+    _activeSetups.clear();
+  });


Two lifecycle edge cases around close():

In-flight _runSetup vs close(): _activeSetups is only populated once setUp() finishes (line 122). If Runner.close() runs while _runSetup is still executing (e.g. Ctrl-C, --stop-on-first-failure, or a test timeout), close() sees _activeSetups empty and completes immediately; if setUp() finishes afterward, it appends to _activeSetups after close() has already run, so its teardowns never run and its isolate is never killed (and while setUp() is running, the open RawReceivePorts keep the VM alive). Tracking/closing in-flight setups or tearing down late completions when _closeMemo.hasRun would avoid leaking them.

Unexpected isolate exit before close(): errorPort is closed as soon as setUp() returns (line 153) and there is no onExit listener (unlike spawn_hybrid.dart). If a background server in the setup isolate crashes with an unhandled async error or calls Isolate.exit(), close() sends 'teardown' to a dead commandPort and waits the full 30-second timeout before reporting Teardown timed out.

Addressed both:

Tracked in-flight setups so close() aborts in-flight isolates/ports and tears down any late completions if _closeMemo.hasRun is true.

Added an onExit port and kept errorPort/exitPort open when hasTearDown is true so unexpected isolate exits before or during teardown fail immediately instead of hitting the 30s timeout.

natebosch · 2026-11-05T23:52:11Z

+    if (_runCalled) {
+      await _suiteController.close();
+    }


Is this if (_runCalled) check left over from an earlier iteration of the PR? If nothing in the current design needs it, let's revert this change to engine.dart.

natebosch · 2026-11-05T23:52:14Z

+        d.file('test_test.dart', '''
+        import 'package:test/scaffolding.dart';
+        import 'package:test/test.dart';
+
+        void main() {
+          test("concurrent call 1", () async {
+            final result = await globalSetup(Uri.parse('/test/slow_setup.dart'));
+            expect(result, equals(1));
+          });
+
+          test("concurrent call 2", () async {
+            final result = await globalSetup(Uri.parse('/test/slow_setup.dart'));
+            expect(result, equals(1));
+          });
+
+          test("concurrent call 3", () async {
+            final result = await globalSetup(Uri.parse('/test/slow_setup.dart'));
+            expect(result, equals(1));
+          });
+        }
+      '''),
+      ]).create();
+
+      var test = await runTest(['test/test_test.dart'], concurrency: 3);


Tests within a single suite run sequentially even with concurrency: 3, so the second and third globalSetup calls here only run after the first has already completed. Put these in separate suite files (or use Future.wait inside a test) so concurrent in-flight globalSetup deduplication is exercised.

Updated to run 3 separate suite files with concurrency: 3 (and Future.wait inside the first suite).

…d tests

Revisions updated by `dart tools/rev_sdk_deps.dart`. core (https://github.com/dart-lang/core/compare/36f57a9..fa0c6b7): fa0c6b77 Fri Nov 6 12:56:58 2026 +0100 Michael Thomsen Release os_detect 2.0.4 (dart-lang/core#1038) dartdoc (https://github.com/dart-lang/dartdoc/compare/03ec6df..7e2508c): 7e2508c6 Wed Nov 11 14:52:22 2026 +0200 Sergey G. Grekhov Add constructor doc comment tests for enums and extension types (dart-lang/dartdoc#4316) 03dfb467 Mon Nov 9 11:00:32 2026 +0200 Sergey G. Grekhov `#3330`. Split test for parameters to separate tests (dart-lang/dartdoc#4314) 23302d53 Mon Nov 9 10:56:34 2026 +0200 Sergey G. Grekhov Add tests for primary constructors (dart-lang/dartdoc#4313) ecosystem (https://github.com/dart-lang/ecosystem/compare/54e1673..1d2b5ff): 1d2b5ff Tue Nov 10 09:04:24 2026 -0800 Kevin Moore feat(repo_manage): add --replace-pattern to report changelog (dart-lang/ecosystem#482) http (https://github.com/dart-lang/http/compare/4646fe6..2bac218): 2bac218 Fri Nov 6 17:49:23 2026 -0800 dart-ecosystem-bot[bot] Tidy: package:cronet_http cdf967c Fri Nov 6 15:40:49 2026 -0800 Brian Quinlan chore(groundkeeper): use stable for flutter (dart-lang/http#2066) 6582177 Thu Nov 5 13:04:35 2026 -0800 Brian Quinlan doc: use timeout as an abort example (dart-lang/http#2062) native (https://github.com/dart-lang/native/compare/6287745..b21ef2d): b21ef2da3 Wed Nov 11 16:07:34 2026 +0530 dhairya05 [json_syntax_generator] Pass the parameter name to ArgumentError.value (dart-lang/native#3787) 20a607e48 Wed Nov 11 07:28:03 2026 -0300 Daco Harkes [code_assets] [hooks] [hooks_runner] Adopt `.g.dart` for generated files (dart-lang/native#3790) cc974ddec Wed Nov 11 19:59:39 2026 +1100 Liam Appelbe [objective_c] Fix `isValidObject` bug on iOS simulator (dart-lang/native#3788) f7edbdee3 Sun Nov 8 13:18:25 2026 +1100 Liam Appelbe [jni] Fix possible receiver NPEs (dart-lang/native#3784) 0ca7d6745 Fri Nov 6 10:20:21 2026 -0300 Daco Harkes [code_assets] [hooks_runner] [native_toolchain_c] Prepare for release (dart-lang/native#3781) baad15ab4 Fri Nov 6 02:56:52 2026 +0100 Kravchenko Igor [objective_c] Link Foundation in the native support library (dart-lang/native#3782) shelf (https://github.com/dart-lang/shelf/compare/9946de5..b175e9f): b175e9f Tue Nov 10 09:14:31 2026 +0530 Muhammad Uvesh shelf_static: ignore invalid If-Modified-Since headers (dart-lang/shelf#557) 02b5f94 Sat Nov 7 23:09:28 2026 +0530 dhairya05 Fix typo in shelf_router docs (dart-lang/shelf#556) test (https://github.com/dart-lang/test/compare/2a9e017..3533ae1): 3533ae1e Tue Nov 10 15:06:08 2026 -0800 Nate Bosch Never retry tearDownAll (dart-lang/test#2815) 5680b0b1 Fri Nov 6 12:52:11 2026 +0100 Sigurd Meldgaard Add globalSetup and addGlobalTearDown APIs for cross-suite setup and teardown (dart-lang/test#2714) b8dc888a Thu Nov 5 18:11:37 2026 -0800 Nate Bosch [checks_codegen] Skip private fields and fields referencing private types (dart-lang/test#2782) 66e062c4 Thu Nov 5 16:05:36 2026 -0800 Nate Bosch [checks_codegen] Generate full field types with imports (dart-lang/test#2781) tools (https://github.com/dart-lang/tools/compare/727fd9c..589d844): 589d844c4 Wed Nov 11 20:27:32 2026 +0300 Hasan Armoush fix(markdown): combine adjacent Text nodes when the element is last in its list (dart-lang/tools#2598) Change-Id: I13acaa7378c86f2c33d88f3c4b163acf2b48ff08 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/576820 Commit-Queue: Nate Bosch <[email protected]> Reviewed-by: Nicholas Shahan <[email protected]>

sigurdm added 4 commits August 5, 2026 14:25

Add tag-scoped pre_run and post_run hooks in dart_test.yaml

f14d5d9

Ensure Engine.close completes _group when engine was not run

fddb75d

Only close suiteController in Engine.close if run was called

3a74fd0

Support post_run hooks under tags and suite configuration

597f6dc

github-actions Bot added the package:test label Aug 5, 2026

sigurdm added 22 commits August 5, 2026 15:10

Update CHANGELOGs and fix analyzer lints

c017999

Support isolated session directory and stale session GC for hooks

b86b614

Fix Engine.close _closedBeforeDone handling and add 24h stale session…

8aed40e

… fallback

Only close _suiteController when _runCalled in Engine.close

0589d29

Expose testSessionPath top-level getter for test session directory

52ec319

Restrict pre_run and post_run hooks to Dart scripts

c52c63c

Address review feedback on hook docs and remove Hook.toString

75a65c5

Support transitive add_tags, real-time output streaming, and session …

740f208

…cleanup grace period

Clarify that hook scripts must exit with code 0

5f03edf

Report post_run hook failures and exit with non-zero exit code

b210be3

Remove unused pre_run_hook.dart

bb9458f

Expose TestEnvironment.sessionDirectory instead of testSessionPath

696d239

Add tryDeleteDirectory helper and use explicit IOException/FormatExce…

9ca13d5

…ption catch blocks

Compile hook scripts to kernel with shared TestCompiler before execution

257e774

Execute post-run teardowns before closing the compiler and loader

97a303b

Add test_environment_io.dart and test_environment_stub.dart

576ebb3

Execute hook scripts directly via Process.start and keep TestCompiler…

2136be2

… isolate-focused

Update syntax error hook test to expect ApplicationException exit code

35d2309

Run hooks in isolates compiled via shared TestCompiler with VmTestTyp…

09c4929

…e.hook

Document and test top-level pre_run and post_run hooks in dart_test.yaml

4326c80

Replace tag hooks and session dirs with globalSetup and addGlobalTear…

567f1d8

…Down APIs

Document globalSetup and addGlobalTearDown in README.md

c63cf32

sigurdm changed the title ~~Add pre_run and post_run hooks to dart_test.yaml~~ Add globalSetup and addGlobalTearDown APIs for cross-suite setup and teardown Aug 10, 2026

Remove pre_run and post_run from configuration.md table of contents

378e0c4

lrhn reviewed Sep 4, 2026

View reviewed changes

Comment thread pkgs/test_api/lib/src/scaffolding/global_setup.dart Resolved

Clarify in README that global setup runs once per URI

ea1b810

lrhn reviewed Sep 4, 2026

View reviewed changes

Comment thread pkgs/test_api/lib/src/scaffolding/global_setup.dart Resolved

Use pattern matching switch in globalSetup and fail on unexpected mes…

078ef88

…sages

lrhn approved these changes Sep 4, 2026

View reviewed changes

Address review comments on global setup and metadata parsing

3076b04

natebosch requested changes Sep 18, 2026

View reviewed changes

natebosch reviewed Sep 18, 2026

View reviewed changes

Comment thread pkgs/test/README.md Resolved

sigurdm added 3 commits September 21, 2026 11:40

Merge branch 'master' into tag-hooks

4c26e0b

Address review feedback: add package:test/global.dart, zone-scope man…

0254e00

…ager, and validate JSON return

Make GlobalSetupManager purely zone-scoped via asCurrent

5f322de

natebosch requested changes Sep 22, 2026

View reviewed changes

sigurdm and others added 5 commits September 25, 2026 12:53

Address review feedback: cache encoded JSON, resolve standalone URIs …

45b0a6e

…to script, and clean up exports

Restore full scaffolding exports in test_api/test_core and use Uri fo…

89ffcdd

…r root-relative check

Use assert and comment for RFC 3986 root-relative path normalization

6d008fd

Merge branch 'master' into tag-hooks

18b0ee3

Nit cleanup

311bdb7

- Reflow long lines - Remove a src import - Remove an unnecessary type on the left

natebosch mentioned this pull request Oct 9, 2026

Plumb experiment flags from analysis_options.yaml to parser. dart-lang/dart_style#1909

Merged

natebosch reviewed Oct 16, 2026

View reviewed changes

Comment thread pkgs/test_core/lib/src/scaffolding.dart Resolved

sigurdm added 3 commits October 20, 2026 11:57

Merge branch 'master' into tag-hooks

8ed5c64

Conditionally import standalone_global_setup.dart and update changelogs

1389c4f

Merge branch 'master' into tag-hooks

ef6c5c3

sigurdm requested a review from natebosch November 2, 2026 06:57

natebosch approved these changes Nov 6, 2026

View reviewed changes

Address review feedback on globalSetup bootstrap, close lifecycle, an…

6ca48c7

…d tests

sigurdm merged commit 5680b0b into master Nov 6, 2026
70 checks passed

sigurdm deleted the tag-hooks branch November 6, 2026 11:52

Conversation

sigurdm commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example

Key Features

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Health

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sigurdm commented Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

natebosch commented Oct 8, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

sigurdm commented Aug 5, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading