Use coverage.collect's coverableLineCache param to speed up coverage
#136851
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
tool
github-actions
bot
added
a: tests
liamappelbe
changed the title
coverage.collect's coverableLineCache param to speed up coveragecoverage.collect's coverableLineCache param to speed up coverage
Contributor
|
wow, impressive benchmarks! |
christopherfujino
approved these changes
Oct 19, 2023
Contributor
christopherfujino
left a comment
christopherfujino
left a comment
There was a problem hiding this comment.
LGTM. btw, this PR rolls package:coverage, although if that merges first I wouldn't think there would be a conflict and this should still merge cleanly as is: https://github.com/flutter/flutter/pull/136924/files
liamappelbe
added
the
autosubmit
| @visibleForTesting bool forceSequential = false, | ||
| @visibleForTesting FlutterVmService? serviceOverride, | ||
| bool branchCoverage = false, | ||
| required Map<String, Set<int>> coverableLineCache, |
Contributor
There was a problem hiding this comment.
@liamappelbe marking this required is a breaking change, and will be difficult to get into Google. Can we mark it as optional for the short term?
auto-submit
bot
removed
the
revert
auto-submit bot
added a commit
that referenced
this pull request
Oct 24, 2023
… up coverage" (#137121) Reverts #136851 Initiated by: CaseyHillers This change reverts the following previous change: Original Description: One of the reasons gathering coverage information is expensive is that we have to force compile every function in the libraries we're interested in. Without this, functions that haven't been invoked (so haven't been compiled) won't have any line number information, so the coverage tool doesn't know which lines to add to the list of misses. In flutter's case, the test infra spawns many VMs, and each of these needs to recompile all those libraries. To fix this, we need a way of skipping force compilation for libraries we've already seen in previous tests, without losing the information about which lines in each library are coverable. So I [added](dart-archive/coverage#466) the `coverableLineCache` to `coverage.collect` in package:coverage v1.7.0. This cache starts out empty, but fills up with lists of all the lines that are coverable for every library as coverage is gathered. package:coverage can then tell the VM not to force compile any libraries in this cache (using `getSourceReport`'s `librariesAlreadyCompiled` param). So the first test suite will still have to compile everything, but subsequent test suites will be much faster. This speeds up coverage collection significantly, for large test suites: | Running flutter/packages/flutter tests... | Time | Overhead | | --- | --- | --- | | without coverage | 8:53 | - | | with coverage | 20:25 | 130% | | with `coverableLineCache` | 12:21 | 40% | Bug: #100751
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 24, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 24, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 24, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 25, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 25, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 25, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Oct 25, 2023
ksokolovskyi
pushed a commit
to ksokolovskyi/flutter
that referenced
this pull request
Oct 31, 2023
…7385) Relands flutter#136851, which was rolled back in flutter#137121 package:coverage has been rolled, so the breakages should be fixed. Also, in this reland I've changed the `coverableLineCache` parameter to be optional, which is safer.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of the reasons gathering coverage information is expensive is that we have to force compile every function in the libraries we're interested in. Without this, functions that haven't been invoked (so haven't been compiled) won't have any line number information, so the coverage tool doesn't know which lines to add to the list of misses. In flutter's case, the test infra spawns many VMs, and each of these needs to recompile all those libraries.
To fix this, we need a way of skipping force compilation for libraries we've already seen in previous tests, without losing the information about which lines in each library are coverable. So I added the
coverableLineCachetocoverage.collectin package:coverage v1.7.0. This cache starts out empty, but fills up with lists of all the lines that are coverable for every library as coverage is gathered. package:coverage can then tell the VM not to force compile any libraries in this cache (usinggetSourceReport'slibrariesAlreadyCompiledparam). So the first test suite will still have to compile everything, but subsequent test suites will be much faster.This speeds up coverage collection significantly, for large test suites:
coverableLineCacheBug: #100751