Msbuild 15.0 hangs when triggered from Powershell

I am trying to upgrade from MSBduil 4.0 to 15.0. The same set of solution files I build take about 9 minutes in total with msbuild 4.0 but take 10 minutes each when I run them with msbuild 15.0. The symptoms are always the same, the build itself (according to the logs) is done within seconds but the process from Start-Process msbuild.exe takes 10minutes to exit.

I tried several suggestions on the web; like using nr:false, /p:UseSharedCompilation=false but nothing worked all the way.

The strange thing is that I run the same msbuild.exe command in cmd and it doesn't reproduce this.

I should mention that when I build multiple solutions back to back, /p:UseSharedCompilation=false seems to work but I also have to use powershell to start cmd.exe which starts msbuild.exe.

If I used /p:UseSharedCompilation=false while building 1 solution only, it takes about double the time it would normally take.

In summary, what I want to do is:

Start-Process $msBuildPath $msBuildArguments
where $msBuildArguments are: /p:Configuration=Debug /m /p:BuildInParallel=true /nr:False "/p:WarningLevel=4" /p:Platform="Any CPU" /target:Rebuild /nologo /v:q /fileLogger "/fileLoggerParameters:LogFile=C:\temp\BuildLogs<solutionName>.msbuild.log;verbosity=normal;ShowTimeStamp"

After a week or so of trying different things, I wanted to ask you for suggestions about what I can do to avoid this.

It sounds like Start-Process isn't returning until all child processes of the launched msbuild.exe exit.

MSBuild spawns other processes, some of which are server processes that persist in an attempt to amortize startup time across many builds. Any task may do this, but there are two processes that commonly persist:

msbuild.exe itself (for multi-process builds)
- You're already disabling this behavior by passing /nodeReuse:false on the command line
vbcscompiler.exe (which does all of the C# and Visual Basic compilation)
- This is not started when you set UseSharedCompilation=false

I don't think there's a way to control the lifetime or timeout of vbcscompiler.exe--there's no way for it to know when "the build" is complete, because MSBuild doesn't really know about it. Only the Csc and Vbc tasks do.

As you've discovered, disabling the compiler server has a performance cost, because you have to wait for csc.exe to initialize for every project, instead of just once.

I can see a few options:

Convince PowerShell to wait only for the top-level child process, and not all of its transitive children
- This should be safe since vbcscompiler.exe will only respond to requests from "compatible" Csc task invocations, so even if the next build switches compiler versions, the process shouldn't harm builds, and at idle it shouldn't be particularly resource-intensive.
Disable the compiler server and live with the longer times.
Some kind of awful attempt to detect MSBuild completion and kill child process.
- I don't recommend this.

I am encountering the same issue and my workaround as of now is to use the default /maxcpucount:1 instead of /m as it doesn't spawn any other process.

@andremakram

@andremakram an alternative might be to add a custom target that will shutdown the vbcscompiler at the end of your build, that way you will not need to wait for it to timeout and shutdown.

dotnet/roslyn#7930 implemented a -shutdown option for the vbcscompiler server

@rainersigwald

Convince PowerShell to wait only for the top-level child process, and not all of its transitive children

@rainersigwald That would be great. How???

Disable the compiler server and live with the longer times.

Limited effectiveness. Still hangs sometimes. I think it hangs on first run after startup or after killing all msbuild.exe.

Some kind of awful attempt to detect MSBuild completion and kill child process

Like poll for completion in powershell? how to know when it's done? Or kill all msbuild.exe from inside each build. yuk

Is there any progress on this issue? I'm facing it right now and it is quite annoying :(

@klinki

@klinki : yes, I can report progress. I did solve the issue some time ago. Thanks for the prompt to share.

Thanks to @rainersigwald for providing ideas to try. Disabling the server aspect doesn't seem possible. I tried several different ways, but was not able to completely turn it off. Killing the child processes might be possible, but would be complicated code and a huge nasty hack. But, waiting for the top-level process to exit turned out to be easy enough and very effective. And, you still get the benefit of the process caching that msbuild provides!

The general idea is to get the process ID from Start-Process and then use Wait-Process:

  $process = Start-Process -FilePath $filePath -ArgumentList $argumentText -NoNewWindow -PassThru
  Wait-Process -InputObject $process

I think it's unfortunate that Start-Process -Wait is confusing. At first blush it seems to provide what you want, but since it's too helpful (waiting for decendents), it's problematic. IMO, the default wait behavior should wait for only the process it creates. It could provide the extra feature of waiting for all decendents via a different switch.

Here's more code to show how to use and integrate the use of Wait-Process.

<#
.SYNOPSIS
Starts a process and waits for it to exit.
.NOTES
The behavior of this is different than using 'Start-Process -Wait' since that waits for the 
process and all _decendent_ processes to exit which causes a hang if the process starts a child 
process that does not exit when the first process exits (something that msbuild often does).

Although this function is only _needed_ if the started process might start long-running child
processes, it is general purpose and without known disadvantages such that it can be used for 
any synchronous, job-like application -- for any situation that you otherwise might use 
'Start-Process -Wait'.
#>
function StartAndWaitForExit([string]$filePath, [string]$argumentText) {
  Write-Host "Run: $filePath $argumentText"
  if (!$argumentText) {
    # Start-Process fails if ArgumentList param is empty, so pass something innocuous
    $argumentText = " "
  }
  $process = Start-Process -FilePath $filePath -ArgumentList $argumentText -NoNewWindow -PassThru
  Wait-Process -InputObject $process
  if ($process.ExitCode -gt 0) {Write-Error "$filePath returned error code $($process.ExitCode)"}
}

<#
.NOTES
The script hangs when invoking msbuild via Start-Process -Wait since it waits for the process and
all decendent processes to exit and msbuild starts and leaves running child processes. Note that 
/nodeReuse:false is supposed to prevent this problem but does not. 
#>
function RunMsBuild([string]$filePath, [string]$additionalArgs="") {
  $argumentText = "$filePath /t:Build `"/p:Configuration=$configuration`" `"/p:Platform=$platform`" /MaxCpuCount $additionalArgs"
  StartAndWaitForExit "msbuild" $argumentText
}

Thanks! it looks good, I will try it. For now, as a workaround, I execute the msbuild from cmd.exe which seems to be working just fine. It's pretty weird, but it works.

You are running cmd.exe with params to run msbuild? Something like this?

Start-Process -Wait -FilePath cmd.exe -ArgumentList "/c msbuild ..."

I found that that works some times. It does not work when no cached msbuild processes are running -- like after reboot. When no cached processes exist, then running msbuild will result in a msbuild sub-process that outlives the creating process and this causes 'Start-Process -Wait' to hang.

But, when a cached process does exist, then when you run msbuild it will not create a new cached one ... and not hang. ... This got me since while testing this stuff. I invariably would end up having a msbuild cached process and then code as you describe works fine. But, I would find that sometimes it didn't (it would hang) and that was very hard to pin down since it was only the first run after reboot.

To test whether your powershell code is working, kill all the msbuild processes first.

You might want to switch nodereuse off in Msbuild arguments, see also https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-command-line-reference?view=vs-2019#switches

I did try setting nodeResult to false. It didn't help me. IOW it would still hang.

Maybe I did something wrong since it seemed to ignore my request to not cache processes.

But ... I am also concerned about performance. msbuild caches processes for performance. If you tell it not to do that (and it works) then you lose that performance gain. I want the performance gain and I want it to not hang.

You are running cmd.exe with params to run msbuild? Something like this?

Start-Process -Wait -FilePath cmd.exe -ArgumentList "/c msbuild ..."

I found that that works some times. It does not work when no cached msbuild processes are running -- like after reboot. When no cached processes exist, then running msbuild will result in a msbuild sub-process that outlives the creating process and this causes 'Start-Process -Wait' to hang.

I execute it like this:

cmd.exe /C """$msBuildPath"" EmSite\\EmSite.csproj /nr:false /m /p:DeployOnBuild=true /p:PublishProfile=FolderProfile";

and it seems to be working fine.

@klinki

@klinki : Seems that you are not using Start-Process. That was not clear from your earlier post ... partly since other earlier posts were exclusively about Start-Process and since you didn't say you were not using start-process. Anyway, if that works, then great. I do wonder if there are any limitations with that approach. Sadly, I'm not working with that script (or company) any more so I'm may not get around to looking into it more.

@stevebroshar

@stevebroshar Yeah, you are right, sorry I wasn't clear about it from the beginning. In ideal world, I would use Start-Process with MSBuild directly, but since it is not working properly, I found this workaround with using cmd.exe and it seems to be working quite well. I never encountered that bug with MSBuild hanging anymore, so for me this workarounds solves that issue.

I can confirm that this problem still remains in the latest build tools.

This problem was haunting me for months and cost me many days of my life while I've been waiting for MSBuild to finish. Anyway... Here is my solution (with UserShareCompilation=false):

Working MSBuild setup

[System.IO.FileInfo] $mapsSolutionFile = 'PathToYourSolutionFile'
nuget restore "$($mapsSolutionFile.FullName)"
[System.IO.FileInfo] $msbuildAbsolutePath = Get-MSBuildAbsolutePath
$msbuildArgumentList = @(
	"`"$($mapsSolutionFile.FullName)`"",
	"/p:Configuration=Release-Dev",
	"/p:Platform=""Any CPU""",
	"/p:UseSharedCompilation=false",
	"/nodeReuse:false"
)
Start-Process -FilePath $msbuildAbsolutePath.FullName -ArgumentList $msbuildArgumentList -NoNewWindow -Wait

function Get-MSBuildAbsolutePath
{
    [OutputType([System.IO.FileInfo])]

    $msbuildAbsolutePath = &"${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -latest -prerelease -products * -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe
    if ([object]::Equals($msbuildAbsolutePath, $null))
    {
        $msbuildAbsolutePath = Get-ChildItem -Path ${env:ProgramFiles(x86)} -File -Recurse -Filter MSBuild.exe -ErrorAction:SilentlyContinue | Select-Object -First 1 -ExpandProperty FullName
    }
    return $msbuildAbsolutePath
}

@KUTlime

@KUTlime I think your solution prevents the temporary hang by disabling process caching. I see how that significantly improves performance. So, it's a solid solution.

The down side is that it turns off the process caching optimization. Also, in my opinion, the problem is that PowerShell waits for sub-processes to exit -- which is just weird.

Just wanting to point out that there is a way to prevent the temporary hang without disabling the process caching optimization -- by using the PowerShell commands in comment above.

@stevebroshar

@stevebroshar I tested both approaches before I wrote my previous comment. Unfortunately, your suggestion with -PassThru and Wait-Process doesn't work on my side (I'm using MSBuild tools 16.9 + PowerShell 7.1.2).

I had this exact issue and it was solved by creating a process using the system. Diagnostic.process class instead of using start-process

Hope this helps at least 1 person

Navigation Menu

Repository navigation

Msbuild 15.0 hangs when triggered from Powershell #2269

Working MSBuild setup

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Msbuild 15.0 hangs when triggered from Powershell #2269

Description

Activity

rainersigwald commented on Jul 7, 2017

superhoang commented on Aug 1, 2017

japj commented on Nov 19, 2017

stevebroshar commented on Jul 21, 2020

klinki commented on Sep 11, 2020

stevebroshar commented on Sep 11, 2020

klinki commented on Sep 14, 2020

stevebroshar commented on Sep 14, 2020

japj commented on Sep 14, 2020

stevebroshar commented on Sep 14, 2020

klinki commented on Oct 13, 2020

stevebroshar commented on Oct 13, 2020

klinki commented on Oct 14, 2020

KUTlime commented on Mar 10, 2021

Working MSBuild setup

stevebroshar commented on Mar 11, 2021

KUTlime commented on Mar 11, 2021

L3Prichaun13 commented on Nov 17, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions