Enable parallel builds across repos of VMR#18824
Conversation
mmitche
left a comment
mmitche
left a comment
There was a problem hiding this comment.
Surprised to see so little benefit compared to my earlier experiments. Let's see how it plays out as the last few repos get enabled.
There was a problem hiding this comment.
Now with repositories running in parallel, how does the console output look like when building locally? I would expect it to not be readable anymore. I wonder if we should re-enable MinimalConsoleLogOutput (we disabled that in the past) with parallel builds enabled.
I wonder if a sequential build with full logging (current experience) makes more sense locally.
Yes, it can be jumbled. Here's a good example snippet from the output: I'll investigate the use of |
The thing is, we intentionally disabled the minimal console log output feature when building locally, a few months ago. I'm not sure if we want to revert that for the sake of parallel build support given that the gains aren't significant on your machine. Even with minimal console log, the log could theoretically still be jumbled. That said, I'm not against enabling it but I wonder if there's a way to prevent the jumbled output. |
|
What was the motivation for disabling minimal output? Just to provide more info? |
Couple of things:
cc @mmitche in case you remember cc @rainersigwald in case you have suggestions / opinions here. Minimal console log output means that output from an msbuild Exec task is redirected into a separate file. Now that we want to enable parallel repo builds, we wonder if we should re-enable that minimal console log feature of if there's something else that we could do in terms of logging. |
|
I'll just get this merged for now and we can follow up later with any perceived usability issues with the log. |
|
Got an interesting build error: I think what's happening here is that two repos that are being built in parallel are both attempting to copy the restored installer/src/SourceBuild/content/repo-projects/Directory.Build.targets Lines 463 to 466 in a246c1d I think this can be resolved with just configuring retries on the |
I don't think we have any great options for you other than those right now. cc @baronfel |
4 participants
Fixes dotnet/source-build#3608
This updates the VMR build configuration so that the repos are built in parallel according to their dependencies. This means that two or more repos with no dependencies between them, such as xdt and symreader, can be built at the same time. Note that the previous configuration of building repos sequentially was already making fairly efficient use of available resources (CPU/disk) so the benefit gained from introducing parallelism is marginal. This will be explained with numbers further below.
By default, the configuration will have parallelism enabled. It can be disabled with
/p:BuildInParallel=false. One side effect of parallelism is the behavior that occurs when an error occurs in the build. For CI builds, you'd want to collect errors that may occur from all the repo builds that may be happening in parallel so that they can be examined afterwards. For dev builds, it's preferable to fail the build on the first error. Otherwise, the build may continue building other repos that are not dependent on the one that failed and just wasting time. There's already theStopOnFirstFailuresetting that is set to true when building the repos but, by default, this is implicitly ignored whenBuildInParallelis true as shown in this output from MSBuild:Therefore, in order to handle this,
RunEachTargetSeparatelyis set to true. But that is only done for dev builds. For CI builds, we essentially wantStopOnFirstFailureto be ignored.Now onto the numbers. As mentioned, enabling parallelism only provides marginal benefit. Build times for source build show the most benefit compared to the vertical builds but that may be because not all repos are yet enabled for vertical builds. Why is there only a small benefit? Because the machine resources are already being used efficiently, running the repo builds in parallel is trying to make use of additional resources that just aren't there. So it ends up taking each of those repos longer to build in total time than it did when being run in a sequential manner. Here are some example build times of repos showing their build time when being built sequentially vs in parallel:
This leads to marginally better build times or even just equivalent build times compared to the original sequential behavior. For total time, this can get source build from a time of 1:12:00 down to 1:06:00, shaving off 6 minutes in the build. For vertical builds, it varies according to the platform. It shows an improvement of anywhere from 0 to 4 mins.