feat(coverage): support web coverage reports in test_with_coverage by kevmoo · Pull Request #2448 · dart-lang/tools

kevmoo · 2026-07-01T02:03:55Z

Add support for web (V8 / Chrome) coverage collection, parsing, and formatting in package:coverage — additively. VM runs keep the existing collection flow.

Add a -p / --platform flag to test_with_coverage. Web (chrome) runs delegate to dart test --coverage; VM runs (the default, and --function-coverage) continue to use the existing VM-service collection flow.
Reject what cannot work rather than silently degrading: platforms other than vm / chrome, --function-coverage or --branch-coverage combined with a web platform, and platform flags passed through to the test script (-p chrome -- -p vm previously ran both platforms).
Web runs report library coverage only by default, matching VM semantics; a new --include-test-files flag opts in to coverage for test files and other non-library sources. Warn instead of writing a silent empty report when nothing survives filtering.
Throw a FormatException in HitMap.parseFiles for files that are not a valid {"coverage": [...]} report (instead of silently ignoring them), and validate scriptId in parseChromeCoverage.
Pre-flight package:test verification and file: URI normalization via package:package_config.
Fix a pre-existing hang: test_with_coverage now fails immediately, propagating the test process's exit code, when that process exits before the VM service is ready (e.g. a --test path that does not exist). Previously it waited forever.

Caveat worth knowing when reviewing: web coverage is measured on compiled JavaScript, so its line counts and percentages are not comparable to VM coverage — code the compiler tree-shakes cannot be reported as uncovered, and inlined code may not map back to lib/. Measured on one package: VM 72.2% vs Chrome 100%; on this repo's own fixture the comparison inverts. --fail-under thresholds must therefore be re-tuned per platform. This is documented in --help and the CHANGELOG.

Unified VM + web reporting in a single lcov — the part of #2434 that asks to unify web coverage alongside VM coverage — is not included here; this PR adds web as a separate mode. Worth knowing for whoever picks that up: merging requires running both platforms through dart test --coverage, which produces no function coverage even on the VM, so unified mode would be line-coverage only. The merge semantics also need a deliberate decision, since a line covered on the VM but tree-shaken out of the web build would read as covered.

Towards #2434

Add support for web (V8 / Chrome) coverage collection, parsing, and formatting in package:coverage. - Delegate test execution in test_with_coverage.dart to dart test --coverage with -p / --platform support. - Auto-detect raw Chrome V8 coverage JSON lists in HitMap.parseFiles. - Add pre-flight package:test verification and file: URI normalization via package:package_config. - Add explicit line-hit assertions in integration and unit tests. - Update GitHub Actions workflow to run VM and Chrome tests in CI. Fixes #2434

github-actions · 2026-07-01T02:04:34Z

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:coverage ⭐	1.16.0-wip	WIP (no publish necessary)

19 already published.
22 WIP (no publish necessary).

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

gemini-code-assist

Code Review

This pull request adds support for collecting and formatting Web (V8 / Chrome) coverage reports in test_with_coverage via a new -p / --platform flag, and adds support for parsing raw Chrome V8 coverage JSON lists in HitMap.parseFiles. The review feedback highlights several critical issues: the raw Chrome V8 precise coverage JSON list parsing path is currently non-functional because the source and source map providers are hardcoded to return null; if no coverage files are found, the output JSON is not written, which will cause subsequent formatting steps to crash; the workspace filtering logic uses a fragile substring check for file: URIs; the new test in chrome_test.dart does not actually exercise the raw Chrome V8 coverage list parsing path; and an accumulator in test_util.dart should be explicitly typed to avoid runtime type errors.

kevmoo · 2026-07-01T02:16:40Z

/gemini review

gemini-code-assist · 2026-07-01T02:32:06Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

… and tests

kevmoo · 2026-07-01T02:37:38Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for collecting and formatting Web (V8 / Chrome) coverage reports in test_with_coverage using the -p/--platform flag, along with parsing raw Chrome V8 coverage JSON lists in HitMap.parseFiles. It also adds pre-flight verification for package:test and URI normalization. Feedback on the changes includes a suggestion to warn users if package_config.json is completely missing (rather than only when package:test is absent), and to simplify a redundant path containment check.

…e path check

kevmoo · 2026-07-01T02:42:31Z

/gemini review

gemini-code-assist

Code Review

This pull request adds support for collecting and formatting Web (V8 / Chrome) coverage reports in test_with_coverage using a new -p / --platform flag. It also enables parsing of raw Chrome V8 coverage JSON lists in HitMap.parseFiles and introduces pre-flight verification using package_config. There are no review comments, so I have no additional feedback to provide.

kevmoo · 2026-07-01T03:00:43Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for collecting and formatting Web (V8 / Chrome) coverage reports via a new --platform flag, along with parsing raw Chrome V8 coverage JSON lists in HitMap.parseFiles. It also adds pre-flight package verification and URI normalization. The review feedback suggests several robustness and performance improvements, including using Uri.tryParse and checking pathSegments to avoid runtime crashes, using .whereType instead of .cast to safely parse JSON lists, and consistently converting scopes to a Set for O(1) lookups.

… hitmap

kevmoo · 2026-07-01T03:07:38Z

/gemini review

github-actions · 2026-07-01T03:10:51Z

PR Health

Coverage

⚠️

File	Coverage
pkgs/coverage/bin/test_with_coverage.dart	💔 Not covered
pkgs/coverage/lib/src/chrome.dart	💚 98 % ⬆️ 0 %
pkgs/coverage/lib/src/collect.dart	💔 86 % ⬇️ 1 %
pkgs/coverage/lib/src/hitmap.dart	💚 92 % ⬆️ 0 %
pkgs/coverage/lib/src/resolver.dart	💚 96 % ⬆️ 0 %
pkgs/coverage/lib/src/util.dart	💚 99 % ⬆️ 0 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

License Headers ✔️

// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/html/test/differential_test.dart
pkgs/html/test/dom_parsing_test.dart
pkgs/html/test/mxss_test.dart
pkgs/html/test/noah_ark_clause_limit_test.dart

This check can be disabled by tagging the PR with skip-license-check.

Breaking changes ✔️

Package	Change	Current Version	New Version	Needed Version	Looking good?
coverage	Non-Breaking	1.15.1	1.16.0-wip	1.16.0-wip	✔️

This check can be disabled by tagging the PR with skip-breaking-check.

Unused Dependencies ✔️

Package	Status
coverage	✔️ All dependencies utilized correctly.

For details on how to fix these, see dependency_validator.

This check can be disabled by tagging the PR with skip-unused-dependencies-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbol	Leaking sources

This check can be disabled by tagging the PR with skip-leaking-check.

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

gemini-code-assist · 2026-07-01T03:22:47Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

…ile lock race

…cleanup

… cleanup

…overage_test

…th_coverage_test

…erge

srawlins · 2026-09-22T18:20:50Z

I don't see anything regarding coverage. Is this just a re-format PR?

kevmoo · 2026-09-22T18:24:14Z

I don't see anything regarding coverage. Is this just a re-format PR?

@srawlins I messed up a precommit hook. fixed now. Sorry!

- Remove unnecessary `HitMap.parseChromeFiles` public API and use `HitMap.parseFiles` directly in `bin/test_with_coverage.dart` (since `dart test --coverage -p chrome` already emits `{"coverage": [...]}` JSON). - Convert `filterHitmapByScope` into `HitMapScopeFilter.filterByScope` extension method on `Map<String, HitMap>`. - Remove leftover Windows `tearDownAll` retry loop in `test/test_with_coverage_test.dart` (handled by `package:test_descriptor`). - Apply Effective Dart doc comment and naming style updates in `lib/src/util.dart` and `test/test_util.dart`.

kevmoo · 2026-10-05T20:34:38Z

Folks?

liamappelbe · 2026-10-06T01:07:52Z

Still waiting for #2448 (comment) and #2448 (comment) to be addressed (_isUriInScope has become ScopedOutput.includesUri but my comment still applies).

liamappelbe

I'm gonna be OOO for the next few weeks, so I'll approve preemptively, assuming my comments are addressed.

…in HitMap - Restore `_dartRun` in `bin/test_with_coverage.dart` and use it across both `_runVmTestsAndCollectCoverage` and `_runWebTestsAndCollectCoverage`. - Remove the web-only `--include-test-files` flag and `HitMapScopeFilter.filterByScope`. - Add `Resolver.toPackageUri` to normalize `file:` URIs inside known package library roots to `package:` URIs and reuse `ScopedOutput.includesScript` directly inside `HitMap.parseJsonSync` (`scopeOutput`).

In `_runWebTestsAndCollectCoverage`, `flags.testScript` defaults to `'test'` (which is the `test:test` executable name in the VM `dart run` flow). Passing it unconditionally as a positional argument to `dart test` caused `dart test` to run the entire `test/` directory alongside any specific test files passed after `--`, loading suites twice in Chrome and failing `RemoteListener.start`.

### Rationale Prepare `package:coverage` `1.16.0` for publishing following the web coverage support added in #2448. ### Summary of Changes - Drop the `-wip` suffix from `version: 1.16.0` in `pkgs/coverage/pubspec.yaml` and `pkgs/coverage/CHANGELOG.md`. - Polish the `1.16.0` changelog entries: order headline features first, clarify early-exit handling in `test_with_coverage` and `FormatException` validation in `parseChromeCoverage`, wrap prose lines within 80 columns, and place the `Require Dart 3.9.` SDK constraint note at the end of the release block. ### Verification - `dart analyze --fatal-infos` (`No issues found!`) - `dart format --output=none --set-exit-if-changed .` (`0 changed`) - `dart pub publish --dry-run` (`Package has 0 warnings.`)

Revisions updated by `dart tools/rev_sdk_deps.dart`. dartdoc (https://github.com/dart-lang/dartdoc/compare/5eb0f2c..e1859cf): e1859cfc Fri Oct 9 14:53:01 2026 +0200 Sarah Zakarias Link a bare prefix only when it maps to exactly one library (dart-lang/dartdoc#4295) c48f6fb9 Fri Oct 9 14:25:42 2026 +0200 Sarah Zakarias Resolve prefixes through the analyzer's PrefixElement.scopeLibraries (dart-lang/dartdoc#4294) 3a6297e2 Wed Oct 7 12:56:14 2026 +0200 Sarah Zakarias Require analyzer 14.5.0 and bump to 9.0.10 (dart-lang/dartdoc#4292) ecosystem (https://github.com/dart-lang/ecosystem/compare/16dca7a..622ef61): 622ef61 Mon Oct 12 00:23:27 2026 -0700 Kevin Moore fix(firehose): enforce -wip on published versions and tighten changelog checks (dart-lang/ecosystem#467) 5576a85 Wed Oct 7 09:29:44 2026 +0200 Moritz Support use-flutter option in groundskeeper workflow (dart-lang/ecosystem#444) http (https://github.com/dart-lang/http/compare/585d433..23f8574): 23f8574 Thu Oct 8 23:27:25 2026 +0700 Cong Bang feat(http2): configurable receive windows and frame size (dart-lang/http#1999) 18051a6 Thu Oct 8 10:39:38 2026 +0200 Moritz Enable Flutter in Groundskeeper workflow for Flutter packages (dart-lang/http#2004) native (https://github.com/dart-lang/native/compare/a2a6179..10515af): 10515afca Mon Oct 12 12:21:52 2026 +0200 Daco Harkes [native_toolchain_c] Do not pass linker flags when compiling static libraries (dart-lang/native#3731) native is held back at 10515af, before dart-lang/native#3728, because it breaks the dartdev `dart install` native assets tests. See dart-lang/native#3743. test (https://github.com/dart-lang/test/compare/d749b10..30acb69): 30acb69e Mon Oct 12 12:42:54 2026 -0700 Nate Bosch Time out browser loads just before the suite load timeout (dart-lang/test#2786) 597a0326 Mon Oct 12 12:22:53 2026 -0700 Kevin Moore Fix misplaced changelog entries and remove no-op health check (dart-lang/test#2789) 0b6efee8 Mon Oct 12 12:16:29 2026 -0700 Nate Bosch [checks] Collapse nested subjects in condition descriptions (dart-lang/test#2784) 17c98493 Mon Oct 12 12:08:23 2026 -0700 Nate Bosch Follow symlinks in precompiled browser test directories (dart-lang/test#2787) 1cc813ec Fri Oct 9 11:59:56 2026 -0700 Nate Bosch [checks] Join nested collapsed subjects with "that" (dart-lang/test#2783) ada7b950 Thu Oct 8 12:03:42 2026 -0700 Nate Bosch Prepare to publish (dart-lang/test#2780) c4b165b3 Wed Oct 7 11:29:08 2026 -0700 Kevin Moore [checks] Require package:meta ^1.17.0 (dart-lang/test#2779) 4a363d70 Tue Oct 6 10:46:31 2026 -0700 Nate Bosch Let Chrome choose its remote debugging port (dart-lang/test#2778) 465d66ab Tue Oct 6 09:35:22 2026 -0700 Nate Bosch Keep Chrome stderr in the browser output when debugging (dart-lang/test#2777) tools (https://github.com/dart-lang/tools/compare/a66d517..2b54c71): 2b54c71af Mon Oct 12 10:16:45 2026 +0200 dart-ecosystem-bot[bot] [boring] Update BoringSSL to 3fed4f20 (dart-lang/tools#2655) 2337f995a Sat Oct 10 03:11:40 2026 +0300 Hasan Armoush [json_rpc_2] Add strictProtocolChecks parameter to Client (dart-lang/tools#2555) c65c69ed1 Fri Oct 9 08:54:25 2026 +0200 Moritz [boring] Move package:boring to dart-lang/tools (2/4): FFI helpers from package:webcrypto (dart-lang/tools#2615) a04e45491 Thu Oct 8 21:07:18 2026 -0700 Kevin Moore chore: wire package:api_summary and package:string_scanner PR auto-labeler (dart-lang/tools#2652) 8bdd4b92c Thu Oct 8 09:20:48 2026 +0200 Morgan :) [code_builder] Allow built_collection 6.x. (dart-lang/tools#2645) c16040d35 Wed Oct 7 17:30:08 2026 -0700 Kevin Moore [api_summary] Sort primary barrel first and experimental entry points last (v1.2.0) (dart-lang/tools#2619) b2d0780db Wed Oct 7 11:27:16 2026 -0700 Kevin Moore [stream_channel] Require package:async ^2.6.0 (dart-lang/tools#2644) 3ab7bf9d0 Wed Oct 7 11:35:27 2026 +0200 Moritz [boring] Move package:boring to dart-lang/tools (1/4): vendored BoringSSL, bindings and source build (dart-lang/tools#2614) 29df6348b Tue Oct 6 15:21:44 2026 -0700 Kevin Moore feat(coverage): support web coverage reports in test_with_coverage (dart-lang/tools#2448) Regenerate pkg/{analyzer,analyzer_plugin,analyzer_testing}/api.txt. The rolled package:api_summary v1.2.0 (dart-lang/tools#2619) sorts the primary barrel first and other entry points after it, so entries in these files are reordered. The APIs themselves are unchanged. Change-Id: Ie8ff19c5a981ea8ecee8939c75457986a947e0a7 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/563220 Reviewed-by: Samuel Rawlins <[email protected]> Auto-Submit: Nate Bosch <[email protected]> Reviewed-by: Paul Berry <[email protected]> Commit-Queue: Paul Berry <[email protected]>

kevmoo requested review from a team and liamappelbe as code owners July 1, 2026 02:03

github-actions Bot added type-infra A repository infrastructure change or enhancement package:coverage labels Jul 1, 2026

gemini-code-assist Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread pkgs/coverage/lib/src/hitmap.dart Resolved

Comment thread pkgs/coverage/bin/test_with_coverage.dart Resolved

Comment thread pkgs/coverage/test/chrome_test.dart Resolved

Comment thread pkgs/coverage/test/test_util.dart Resolved

fix(review): fix analyzer issues in test_with_coverage.dart

11a011f

fix(review): address reviewer feedback on hitmap, test_with_coverage,…

310608d

… and tests

gemini-code-assist Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread pkgs/coverage/bin/test_with_coverage.dart Resolved

Comment thread pkgs/coverage/bin/test_with_coverage.dart Resolved

fix(review): warn if package_config.json is missing and simplify scop…

61d4489

…e path check

gemini-code-assist Bot reviewed Jul 1, 2026

View reviewed changes

fix(ci): run dart test --platform vm in coverage workflow

51106a8

gemini-code-assist Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread pkgs/coverage/bin/test_with_coverage.dart Resolved

Comment thread pkgs/coverage/lib/src/hitmap.dart Resolved

Comment thread pkgs/coverage/bin/test_with_coverage.dart Resolved

fix(review): apply Uri.tryParse, pathSegments check, and whereType in…

481eda0

… hitmap

kevmoo added 6 commits June 30, 2026 22:56

ci: trigger fresh workflow run with vm test platform

05c9ee3

test(coverage): add tearDownAll delay on Windows to prevent sandbox f…

8002f95

…ile lock race

test(coverage): add retry loop to tearDownAll on Windows for sandbox …

d51f239

…cleanup

test(coverage): update tearDownAll to delay 3s on Windows for sandbox…

3a89f58

… cleanup

test(coverage): increase timeout for Chrome test to 5m in test_with_c…

7224e56

…overage_test

test(coverage): refine Windows tearDownAll sandbox cleanup in test_wi…

38d073b

…th_coverage_test

github-actions Bot added the package:yaml label Sep 22, 2026

chore: revert pre-commit hook formatting on untouched packages from m…

55a814c

…erge

github-actions Bot removed package:unified_analytics package:json_rpc_2 package:code_builder package:html package:markdown package:yaml labels Sep 22, 2026

kevmoo removed request for a team and srawlins September 22, 2026 17:58

natebosch reviewed Sep 23, 2026

View reviewed changes

kevmoo requested review from liamappelbe and natebosch October 1, 2026 06:48

liamappelbe approved these changes Oct 6, 2026

View reviewed changes

Comment thread pkgs/coverage/lib/src/hitmap.dart Resolved

kevmoo added 2 commits October 6, 2026 21:01

kevmoo merged commit 29df634 into main Oct 6, 2026
27 checks passed

kevmoo deleted the web_coverage branch October 6, 2026 22:21

This was referenced Oct 7, 2026

[web/debugging] Refresh dart.dev/web/debugging for modern Dart 3 web & Wasm workflows dart-lang/site-www#7561

Closed

chore(coverage): prepare for v1.16.0 release #2643

Merged

amanmaurya92 mentioned this pull request Oct 14, 2026

fix(coverage): allow empty coverage files from skipped test suites (#2667) #2668

Merged

Conversation

kevmoo commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Package publishing

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevmoo commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

kevmoo commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

kevmoo commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

kevmoo commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevmoo commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Health

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

srawlins commented Sep 22, 2026

Uh oh!

kevmoo commented Sep 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevmoo commented Oct 5, 2026

Uh oh!

liamappelbe commented Oct 6, 2026

Uh oh!

liamappelbe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kevmoo commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading