Skip to content

Strengthen reviewer skill: add step-back analysis dimensions#13504

Merged
JanProvaznik merged 1 commit into
dotnet:mainfrom
JanProvaznik:skills/reviewer-stepback
Apr 8, 2026
Merged

Strengthen reviewer skill: add step-back analysis dimensions#13504
JanProvaznik merged 1 commit into
dotnet:mainfrom
JanProvaznik:skills/reviewer-stepback

Conversation

@JanProvaznik

Copy link
Copy Markdown
Member

Learnings from reviewing PR #13477 — the 24-dimension checklist missed design-level issues that required stepping back from the diff.

Changes

  • Dim 4 (Tests): Flag weak assertions that would pass with incorrect output
  • Dim 10 (Design): Check alignment with original design intent, flag workarounds when better APIs exist in existing dependencies, validate borrowed patterns
  • Dim 22 (Correctness): Generalize 2-participant fixes to N participants, flag symptom-only fixes
  • Workflow: Add historical context step (read linked issue + original feature PR) before dispatching dimension agents
  • Tasks instructions: Document \Microsoft.IO.Redist\ API availability for .NET Framework

Why

The review agents evaluated the diff mechanically but missed:

  1. The fix patches a TOCTOU symptom; a better API (\Microsoft.IO.File.Move(overwrite: true)) exists in an already-referenced package
  2. The fix handles 2 concurrent writers but fails with 3+
  3. Test assertions were too weak to verify correct behavior
  4. The approach was borrowed from the VS editor (which never has 'file doesn't exist' case) without validating assumptions

Add rules for historical context analysis, root-cause vs symptom fixes,
N-participant race generalization, assertion strength validation, and
API alternative awareness to the expert-reviewer agent.

- Dim 4 (Tests): flag weak assertions that pass with wrong output
- Dim 10 (Design): check design intent alignment, API alternatives,
  and borrowed-pattern assumptions
- Dim 22 (Correctness): generalize N=2 to N=3+ participants, flag
  symptom-only fixes
- Workflow: add historical context step before dimension dispatch
- Tasks instructions: document Microsoft.IO.Redist API availability

Co-authored-by: Copilot <[email protected]>
@JanProvaznik JanProvaznik marked this pull request as ready for review April 8, 2026 11:29
@JanProvaznik JanProvaznik requested a review from a team as a code owner April 8, 2026 11:29
Copilot AI review requested due to automatic review settings April 8, 2026 11:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the repo’s review guidance to help reviewers “step back” from mechanical diff-checking and catch design-level issues (especially around concurrency, TOCTOU patterns, and weak test assertions), incorporating lessons from PR #13477.

Changes:

  • Strengthen the Tests dimension to flag weak assertions that could pass with incorrect behavior.
  • Expand Design and Correctness dimensions to emphasize design-intent alignment, avoiding workaround chains when better APIs exist, validating borrowed patterns, and generalizing race fixes beyond 2 participants.
  • Update workflow to include a historical context step (read linked issue + original feature PR) and document Microsoft.IO.Redist API availability for tasks targeting .NET Framework.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/instructions/tasks.instructions.md Adds guidance to check Microsoft.IO.* backported APIs (via Microsoft.IO.Redist) before implementing workaround-based I/O patterns in tasks.
.github/agents/expert-reviewer.md Extends the reviewer checklist with “step-back” design/correctness checks and adds a historical-context workflow step.

Comment thread .github/agents/expert-reviewer.md
@JanProvaznik JanProvaznik requested a review from JanKrivanek April 8, 2026 11:46
@JanProvaznik JanProvaznik enabled auto-merge (squash) April 8, 2026 13:11
@JanProvaznik JanProvaznik merged commit 225d17f into dotnet:main Apr 8, 2026
14 checks passed
dfederm pushed a commit to dfederm/msbuild that referenced this pull request Apr 9, 2026
…13504)

Learnings from reviewing PR dotnet#13477 — the 24-dimension checklist missed
design-level issues that required stepping back from the diff.

### Changes
- **Dim 4 (Tests)**: Flag weak assertions that would pass with incorrect
output
- **Dim 10 (Design)**: Check alignment with original design intent, flag
workarounds when better APIs exist in existing dependencies, validate
borrowed patterns
- **Dim 22 (Correctness)**: Generalize 2-participant fixes to N
participants, flag symptom-only fixes
- **Workflow**: Add historical context step (read linked issue +
original feature PR) before dispatching dimension agents
- **Tasks instructions**: Document \Microsoft.IO.Redist\ API
availability for .NET Framework

### Why
The review agents evaluated the diff mechanically but missed:
1. The fix patches a TOCTOU symptom; a better API
(\Microsoft.IO.File.Move(overwrite: true)\) exists in an
already-referenced package
2. The fix handles 2 concurrent writers but fails with 3+
3. Test assertions were too weak to verify correct behavior
4. The approach was borrowed from the VS editor (which never has 'file
doesn't exist' case) without validating assumptions

Co-authored-by: Copilot <[email protected]>
github-actions Bot pushed a commit to IntelliTect/Multitool that referenced this pull request Jun 29, 2026
Updated [Microsoft.Build](https://github.com/dotnet/msbuild) from 18.6.3
to 18.7.1.

<details>
<summary>Release notes</summary>

_Sourced from [Microsoft.Build's
releases](https://github.com/dotnet/msbuild/releases)._

## 18.7.1

## What's Changed
* Fix TraceEngine file contention deadlock in multithreaded mode by
@​JanProvaznik in dotnet/msbuild#13446
* Remove duplicate test cases in MultithreadableTaskAnalyzer by
@​Youssef1313 in dotnet/msbuild#13483
* Ensure ThreadSafeTaskAnalyzer.Tests is considered as a unit test
project by @​Youssef1313 in dotnet/msbuild#13481
* Fix MSBuildTask0002 analyzer warnings in already-migrated tasks by
@​JanProvaznik in dotnet/msbuild#13466
* Fix race conditions in task host path resolution by @​AR-May in
dotnet/msbuild#13485
* Migrate ToolTask and Al task to TaskEnvironment API by @​OvesN in
dotnet/msbuild#13423
* Bump main to 18.7, add vs18.6 to merge flow by @​MichalPavlik in
dotnet/msbuild#13472
* Avoid allocations in GetHashCode implementations by @​DustinCampbell
in dotnet/msbuild#13475
* Add PATs rotation to agentic workflow(s) by @​JanKrivanek in
dotnet/msbuild#13496
* Fix ASP.NET WebSite projects to copy netstandard.dll facade when
required by @​JanProvaznik in
dotnet/msbuild#13058
* Migrate AspNetCompiler to TaskEnvironment API by @​OvesN in
dotnet/msbuild#13424
* Add review workflow by @​JanKrivanek in
dotnet/msbuild#13503
* Strengthen reviewer skill: add step-back analysis dimensions by
@​JanProvaznik in dotnet/msbuild#13504
* Add 'Request Speedometer Perf Run' to VS experimental insertion build
policies by @​Copilot in dotnet/msbuild#13505
* Remove duplicate @ prefix from issueAuthor in GitOps by @​akoeplinger
in dotnet/msbuild#13492
* Improve review aw by @​JanKrivanek in
dotnet/msbuild#13510
* Migrates unit tests to use RoslynCodeTaskFactory to enable running
tests under .NET Core by @​jankratochvilcz in
dotnet/msbuild#13500
* Fix cross-AppDomain TaskItem modifier cache regression by
@​DustinCampbell in dotnet/msbuild#13493
* Discourage review agent from approving PRs by @​JanKrivanek in
dotnet/msbuild#13512
* Stop trying to deploy ValueTuple by @​rainersigwald in
dotnet/msbuild#13507
* Ad-hoc re-sign bootstrap dotnet on macOS to prevent SIGKILL by
@​jankratochvilcz in dotnet/msbuild#13513
* RoslynCodeTaskFactory: Log MSB3753 when task class does not implement
ITask by @​jankratochvilcz in
dotnet/msbuild#13517
* Update gh-aw (upon mcp policy changes) by @​JanKrivanek in
dotnet/msbuild#13526
* Eliminate XmlChildNodes allocations in GetXmlNodeInnerContents by
@​nareshjo in dotnet/msbuild#13509
* Fix telemetry allocation regression: per-engine collector ownership by
@​JanProvaznik in dotnet/msbuild#13516
* Migrate to xunit.v3 by @​Youssef1313 in
dotnet/msbuild#13482
* Fix stray brace in HandleBuildCancel trace string causing MSB1025 by
@​Copilot in dotnet/msbuild#13535
* Bumping to 10.0.4 runtime packages by @​MichalPavlik in
dotnet/msbuild#13533
* Remove early return in GetCanonicalForm, always call System.IO.Path by
@​OvesN in dotnet/msbuild#13532
* Do not overwrite GetCopyToOutputDirectoryItemsDependsOn, just add new…
by @​snechaev in dotnet/msbuild#13474
* Migrate GetReferenceAssemblyPaths task to TaskEnvironment API by
@​OvesN in dotnet/msbuild#13495
* Stabilize ToolTaskThatTimeoutAndRetry test by @​rainersigwald in
dotnet/msbuild#13489
* [automated] Merge branch 'vs18.6' => 'main' by @​github-actions[bot]
in dotnet/msbuild#13506
* Add extra test assertions around tests by @​Youssef1313 in
dotnet/msbuild#13536
* Add static eval for repo skills/agents via skill-validator by
@​JanKrivanek in dotnet/msbuild#13537
* Migrate SGen task to Task environment API by @​OvesN in
dotnet/msbuild#13457
* Fix TerminalLogger assert failure for metaproj files and cached
project eval ID by @​OvesN in
dotnet/msbuild#13480
* Filter out approving review from pr-reviewer agent by @​JanKrivanek in
dotnet/msbuild#13553
* Use a unique task name per invocation to tabilize
RoslynCodeTaskFactory_ReuseCompilation test by @​huulinhnguyen-dev in
dotnet/msbuild#13551
* Brief doc on feedback/logging/data systems by @​rainersigwald in
dotnet/msbuild#13554
* Localized file check-in by OneLocBuild Task: Build definition ID 9434:
Build ID 13881982 by @​dotnet-bot in
dotnet/msbuild#13437
* Stage 3: Forward BuildProjectFile* callbacks from OOP TaskHost to
worker node by @​JanProvaznik in
dotnet/msbuild#13350
* Enable TaskHost Callbacks by default by @​JanProvaznik in
dotnet/msbuild#13579
* Remove unactionable info from reviewer agent by @​JanKrivanek in
dotnet/msbuild#13578
* Enlighten RequiresFramework35SP1Assembly task for multithreaded mode
by @​jankratochvilcz in dotnet/msbuild#13575
* Make SdkResolver-provided environment variables take precedence over
ambient environment by @​Copilot in
dotnet/msbuild#12655
* Add dotnet/skills marketplace and enable plugins by @​Evangelink in
dotnet/msbuild#13582
* The skills/agents check filters-in only touched files by @​JanKrivanek
in dotnet/msbuild#13586
* Fix skill-validation workflow failing when agents directory is deleted
by @​JeremyKuhne in dotnet/msbuild#13592
 ... (truncated)

Commits viewable in [compare
view](dotnet/msbuild@v18.6.3...v18.7.1).
</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants