Fix TraceEngine file contention deadlock in multithreaded mode#13446
Merged
Conversation
In multithreaded (-mt) mode, multiple in-proc node engines run in the
same process. Each engine's TraceEngine() method wrote to the same
EngineTrace_{PID}.txt file with FileShare.Read. When two engines traced
concurrently, the second got an IOException, which propagated through
QueueAction's error handler and fatally crashed the ActionBlock work
queue. All subsequent Post() calls were silently rejected, causing
request completions to be dropped and the scheduler to deadlock.
Fix:
- Use per-node trace files: EngineTrace_{PID}_{NodeId}.txt
- Catch IOException in TraceEngine so trace failures can never crash
the build engine
Added regression test that enables MSBUILDDEBUGSCHEDULER=1 during a
multithreaded build with parallel child projects.
Co-authored-by: Copilot <[email protected]>
In -mt mode, multiple in-proc BuildRequestEngine instances share one process.
Each engine's TraceEngine() used lock(this), which only serialized within that
instance. Concurrent FileStream opens to EngineTrace_{PID}.txt with
FileShare.Read caused IOException, which fatally crashed the ActionBlock work
queue and caused a scheduler deadlock.
Fix:
- Replace lock(this) with a static lock (s_traceLock) so all engines in the
same process serialize trace writes to a single file.
- Catch non-critical exceptions in TraceEngine, matching the defensive pattern
already used by Scheduler.TraceScheduler.
- Add regression test with Timeout=30s to fail fast on deadlock.
Co-authored-by: Copilot <[email protected]>
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a multithreaded (-mt) in-proc deadlock path caused by concurrent TraceEngine() writes to a shared per-process trace file, which could fault the ActionBlock work queue and stall the scheduler.
Changes:
- Serialize
BuildRequestEngine.TraceEngine()file writes across all in-proc engine instances via a static lock. - Make
TraceEngine()resilient by swallowing non-critical exceptions during trace emission. - Add a regression unit test that runs an
-mtbuild with debug scheduler tracing enabled to ensure the build completes.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/Build/BackEnd/Components/BuildRequestEngine/BuildRequestEngine.cs | Uses a process-wide static lock for engine trace file writes and prevents trace IO failures from crashing the engine. |
| src/Build.UnitTests/BackEnd/BuildManager_Tests.cs | Adds a regression test targeting the previous -mt trace-file contention/deadlock scenario. |
Co-authored-by: Copilot <[email protected]>
AR-May
approved these changes
Mar 31, 2026
FrameworkDebugUtils.SetDebugPath() is parameterless; it reads from the MSBUILDDEBUGPATH environment variable which is already set on the previous line. The incorrect SetDebugPath(debugPath) call caused CS1501 across all CI legs. Co-authored-by: Copilot <[email protected]>
When a non-critical exception occurs writing the engine trace file, log the failure message and stack trace at Low importance through _nodeLoggingContext (if available) so it appears in diagnostic-level build output. Co-authored-by: Copilot <[email protected]>
xUnit only supports Timeout on async tests. Wrap the test body in Task.Run and return async Task. Use fully-qualified Task type to avoid ambiguity with Microsoft.Build.Utilities.Task. Co-authored-by: Copilot <[email protected]>
AR-May
reviewed
Mar 31, 2026
…gine.cs Co-authored-by: AR-May <[email protected]>
dfederm
pushed a commit
to dfederm/msbuild
that referenced
this pull request
Apr 9, 2026
This was referenced Jun 10, 2026
This was referenced Jun 26, 2026
github-actions Bot
pushed a commit
to IntelliTect/Multitool
that referenced
this pull request
Jun 29, 2026
Updated [Microsoft.Build](https://github.com/dotnet/msbuild) from 18.6.3 to 18.7.1. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Build's releases](https://github.com/dotnet/msbuild/releases)._ ## 18.7.1 ## What's Changed * Fix TraceEngine file contention deadlock in multithreaded mode by @JanProvaznik in dotnet/msbuild#13446 * Remove duplicate test cases in MultithreadableTaskAnalyzer by @Youssef1313 in dotnet/msbuild#13483 * Ensure ThreadSafeTaskAnalyzer.Tests is considered as a unit test project by @Youssef1313 in dotnet/msbuild#13481 * Fix MSBuildTask0002 analyzer warnings in already-migrated tasks by @JanProvaznik in dotnet/msbuild#13466 * Fix race conditions in task host path resolution by @AR-May in dotnet/msbuild#13485 * Migrate ToolTask and Al task to TaskEnvironment API by @OvesN in dotnet/msbuild#13423 * Bump main to 18.7, add vs18.6 to merge flow by @MichalPavlik in dotnet/msbuild#13472 * Avoid allocations in GetHashCode implementations by @DustinCampbell in dotnet/msbuild#13475 * Add PATs rotation to agentic workflow(s) by @JanKrivanek in dotnet/msbuild#13496 * Fix ASP.NET WebSite projects to copy netstandard.dll facade when required by @JanProvaznik in dotnet/msbuild#13058 * Migrate AspNetCompiler to TaskEnvironment API by @OvesN in dotnet/msbuild#13424 * Add review workflow by @JanKrivanek in dotnet/msbuild#13503 * Strengthen reviewer skill: add step-back analysis dimensions by @JanProvaznik in dotnet/msbuild#13504 * Add 'Request Speedometer Perf Run' to VS experimental insertion build policies by @Copilot in dotnet/msbuild#13505 * Remove duplicate @ prefix from issueAuthor in GitOps by @akoeplinger in dotnet/msbuild#13492 * Improve review aw by @JanKrivanek in dotnet/msbuild#13510 * Migrates unit tests to use RoslynCodeTaskFactory to enable running tests under .NET Core by @jankratochvilcz in dotnet/msbuild#13500 * Fix cross-AppDomain TaskItem modifier cache regression by @DustinCampbell in dotnet/msbuild#13493 * Discourage review agent from approving PRs by @JanKrivanek in dotnet/msbuild#13512 * Stop trying to deploy ValueTuple by @rainersigwald in dotnet/msbuild#13507 * Ad-hoc re-sign bootstrap dotnet on macOS to prevent SIGKILL by @jankratochvilcz in dotnet/msbuild#13513 * RoslynCodeTaskFactory: Log MSB3753 when task class does not implement ITask by @jankratochvilcz in dotnet/msbuild#13517 * Update gh-aw (upon mcp policy changes) by @JanKrivanek in dotnet/msbuild#13526 * Eliminate XmlChildNodes allocations in GetXmlNodeInnerContents by @nareshjo in dotnet/msbuild#13509 * Fix telemetry allocation regression: per-engine collector ownership by @JanProvaznik in dotnet/msbuild#13516 * Migrate to xunit.v3 by @Youssef1313 in dotnet/msbuild#13482 * Fix stray brace in HandleBuildCancel trace string causing MSB1025 by @Copilot in dotnet/msbuild#13535 * Bumping to 10.0.4 runtime packages by @MichalPavlik in dotnet/msbuild#13533 * Remove early return in GetCanonicalForm, always call System.IO.Path by @OvesN in dotnet/msbuild#13532 * Do not overwrite GetCopyToOutputDirectoryItemsDependsOn, just add new… by @snechaev in dotnet/msbuild#13474 * Migrate GetReferenceAssemblyPaths task to TaskEnvironment API by @OvesN in dotnet/msbuild#13495 * Stabilize ToolTaskThatTimeoutAndRetry test by @rainersigwald in dotnet/msbuild#13489 * [automated] Merge branch 'vs18.6' => 'main' by @github-actions[bot] in dotnet/msbuild#13506 * Add extra test assertions around tests by @Youssef1313 in dotnet/msbuild#13536 * Add static eval for repo skills/agents via skill-validator by @JanKrivanek in dotnet/msbuild#13537 * Migrate SGen task to Task environment API by @OvesN in dotnet/msbuild#13457 * Fix TerminalLogger assert failure for metaproj files and cached project eval ID by @OvesN in dotnet/msbuild#13480 * Filter out approving review from pr-reviewer agent by @JanKrivanek in dotnet/msbuild#13553 * Use a unique task name per invocation to tabilize RoslynCodeTaskFactory_ReuseCompilation test by @huulinhnguyen-dev in dotnet/msbuild#13551 * Brief doc on feedback/logging/data systems by @rainersigwald in dotnet/msbuild#13554 * Localized file check-in by OneLocBuild Task: Build definition ID 9434: Build ID 13881982 by @dotnet-bot in dotnet/msbuild#13437 * Stage 3: Forward BuildProjectFile* callbacks from OOP TaskHost to worker node by @JanProvaznik in dotnet/msbuild#13350 * Enable TaskHost Callbacks by default by @JanProvaznik in dotnet/msbuild#13579 * Remove unactionable info from reviewer agent by @JanKrivanek in dotnet/msbuild#13578 * Enlighten RequiresFramework35SP1Assembly task for multithreaded mode by @jankratochvilcz in dotnet/msbuild#13575 * Make SdkResolver-provided environment variables take precedence over ambient environment by @Copilot in dotnet/msbuild#12655 * Add dotnet/skills marketplace and enable plugins by @Evangelink in dotnet/msbuild#13582 * The skills/agents check filters-in only touched files by @JanKrivanek in dotnet/msbuild#13586 * Fix skill-validation workflow failing when agents directory is deleted by @JeremyKuhne in dotnet/msbuild#13592 ... (truncated) Commits viewable in [compare view](dotnet/msbuild@v18.6.3...v18.7.1). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In multithreaded (
-mt) mode, multiple in-procBuildRequestEngineinstances run in the same process. Each engine'sTraceEngine()method wrote to the sameEngineTrace_{PID}.txtfile usinglock(this), which only serialized within that single instance. When two engines traced concurrently, the secondFileStreamopen (withFileShare.Read) threwIOException, which propagated throughQueueAction's error handler and fatally crashed theActionBlockwork queue. All subsequentPost()calls were silently rejected, causing request completions to be dropped and the scheduler to deadlock.Root Cause
lock(this)only serializes within ONE engine instance. Cross-instance contention on the shared file was unprotected.Fix
lock(this)with a static lock (s_traceLock) so all engine instances in the same process serialize trace writes to a single file. This keeps one unified trace file (better debugging UX than per-node files).TraceEngine()usingcatch (Exception e) when (!ExceptionHandling.IsCriticalException(e)), matching the defensive pattern already used byScheduler.TraceScheduler(Scheduler.cs:2683). Trace file failures must never crash the build engine.Testing
Added regression test
MultiThreadedBuild_WithDebugSchedulerTracing_DoesNotDeadlock:MSBUILDDEBUGSCHEDULER=1during a multithreaded build with 4 parallel child projects[Fact(Timeout = 30_000)]so a deadlock regression fails fast instead of hanging CIRed-green validated locally: without the fix, the test aborts with
IOException: The process cannot access the file 'EngineTrace_{PID}.txt' because it is being used by another processfollowed byInternalErrorException: Failure during engine shutdown. With the fix, the test passes in ~400ms.