Skip to content

Add static eval for repo skills/agents via skill-validator#13537

Merged
JanKrivanek merged 4 commits into
dotnet:mainfrom
JanKrivanek:dev/jankrivanek/skill-validator
Apr 15, 2026
Merged

Add static eval for repo skills/agents via skill-validator#13537
JanKrivanek merged 4 commits into
dotnet:mainfrom
JanKrivanek:dev/jankrivanek/skill-validator

Conversation

@JanKrivanek

Copy link
Copy Markdown
Member

Context

Let's use the skill-validator to check for basic problems of our skills and agents
This is inspired and sourced from similar integrations:

Changes Made

  • Added pipeline using the skill-validator
  • Fixed the current findings (renames of folders and agent file)

Testing

  • skill-validator is the test harness for the skills

@JanKrivanek JanKrivanek requested a review from a team as a code owner April 14, 2026 16:00
Copilot AI review requested due to automatic review settings April 14, 2026 16:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automated static validation for repository Copilot skills and agents using skill-validator, and updates existing repo references/content to satisfy validator expectations.

Changes:

  • Added a PR/push-triggered workflow to run skill-validator and upload results as an artifact.
  • Added a workflow_run follow-up workflow to post (or update) a PR comment with summarized validator results.
  • Added/updated skills and agent content (including a shared workflow doc reference fix) to align with the validator.
Show a summary per file
File Description
.github/workflows/skill-validation.yml Runs skill-validator on skills/agents, summarizes results, uploads artifact for downstream comment workflow.
.github/workflows/skill-validation-comment.yml Downloads validator artifact and posts/updates a PR comment summarizing findings.
.github/workflows/shared/review-shared.md Updates agent filename reference to .agent.md.
.github/skills/optimizing-msbuild-performance/SKILL.md New performance-focused skill content.
.github/skills/maintaining-binary-log-compatibility/SKILL.md New binlog compatibility skill content.
.github/skills/integrating-sdk-and-msbuild/SKILL.md New SDK↔MSBuild integration skill content.
.github/skills/authoring-errors-and-warnings/SKILL.md New diagnostics authoring skill content.
.github/skills/assessing-breaking-changes/SKILL.md New breaking-change assessment skill content.
.github/skills/changewaves/SKILL.md Fixes skill link target to the new assessing-breaking-changes skill.
.github/agents/expert-reviewer.agent.md Adds/renames the expert reviewer agent definition and guidance.

Copilot's findings

  • Files reviewed: 4/10 changed files
  • Comments generated: 3

Comment thread .github/workflows/skill-validation.yml
Comment thread .github/workflows/skill-validation.yml Outdated
Comment thread .github/workflows/skill-validation-comment.yml

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Renames (items 1-6): Clean. All old directory names have been grepped across the repo — no stale references remain in .md, .yml, or other files. The changewaves/SKILL.md cross-reference fix (item 7) and review-shared.md agent path update (item 8) are correct.

Security model: The workflow_run split between skill-validation.yml (read-only, runs PR code) and skill-validation-comment.yml (write permissions, never checks out PR code) correctly follows the safe fork-PR pattern. ✅

Findings

Severity Finding
⚠️ Hardcoded counts in skill-validation.yml:131-134 — skill-count (13), agent-count (1), total (14) are static and will silently go stale. Should be computed dynamically.
i️ Unpinned nightly binaryskill-validator-nightly tag is mutable; a bad upstream release could break CI. Consider pinning or adding checksum verification.
i️ No graceful handling if artifact upload was skipped — the comment workflow will fail hard if the validation workflow was cancelled before uploading.
i️ Success summary is sparse — on a clean run the step summary just says "All checks passed" with no scope detail.

No breaking changes, no security issues, no stale references found. The hardcoded counts are the main item worth addressing before merge.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #13537 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Expert Code Review (on open) for issue #13537 · ● 2.5M

Comment thread .github/workflows/skill-validation.yml Outdated
Comment thread .github/workflows/skill-validation.yml
Comment thread .github/workflows/skill-validation-comment.yml
Comment thread .github/workflows/skill-validation.yml
@JanKrivanek JanKrivanek marked this pull request as draft April 14, 2026 16:12
@JanKrivanek JanKrivanek marked this pull request as ready for review April 14, 2026 17:22
@JanKrivanek JanKrivanek enabled auto-merge April 14, 2026 18:38
@JanKrivanek JanKrivanek merged commit 41053b8 into dotnet:main Apr 15, 2026
11 of 13 checks passed
@JanKrivanek JanKrivanek deleted the dev/jankrivanek/skill-validator branch April 15, 2026 11:52
JanKrivanek added a commit that referenced this pull request Apr 21, 2026
#13537 followup

### Context
The PR above introduced skills/agents files checking within PRs touching
skills/agents
But it checked all the files - possibly repeating same and unactionable
findings

### Changes Made
We now filter only for the touched skills and agent files
PureWeen pushed a commit to dotnet/maui that referenced this pull request Apr 21, 2026
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Description of Change

Adopting skills-validator

This is inspired and sourced from similar integrations:
 * github/awesome-copilot#1195
 * dotnet/runtime#126814
 * dotnet/msbuild#13537

One important difference is that in this case we run as well in the LLM
judging mode

cc: @PureWeen - for awareness and for help with iterating on this :)
github-actions Bot pushed a commit to IntelliTect/Multitool that referenced this pull request Jun 29, 2026
Updated [Microsoft.Build](https://github.com/dotnet/msbuild) from 18.6.3
to 18.7.1.

<details>
<summary>Release notes</summary>

_Sourced from [Microsoft.Build's
releases](https://github.com/dotnet/msbuild/releases)._

## 18.7.1

## What's Changed
* Fix TraceEngine file contention deadlock in multithreaded mode by
@​JanProvaznik in dotnet/msbuild#13446
* Remove duplicate test cases in MultithreadableTaskAnalyzer by
@​Youssef1313 in dotnet/msbuild#13483
* Ensure ThreadSafeTaskAnalyzer.Tests is considered as a unit test
project by @​Youssef1313 in dotnet/msbuild#13481
* Fix MSBuildTask0002 analyzer warnings in already-migrated tasks by
@​JanProvaznik in dotnet/msbuild#13466
* Fix race conditions in task host path resolution by @​AR-May in
dotnet/msbuild#13485
* Migrate ToolTask and Al task to TaskEnvironment API by @​OvesN in
dotnet/msbuild#13423
* Bump main to 18.7, add vs18.6 to merge flow by @​MichalPavlik in
dotnet/msbuild#13472
* Avoid allocations in GetHashCode implementations by @​DustinCampbell
in dotnet/msbuild#13475
* Add PATs rotation to agentic workflow(s) by @​JanKrivanek in
dotnet/msbuild#13496
* Fix ASP.NET WebSite projects to copy netstandard.dll facade when
required by @​JanProvaznik in
dotnet/msbuild#13058
* Migrate AspNetCompiler to TaskEnvironment API by @​OvesN in
dotnet/msbuild#13424
* Add review workflow by @​JanKrivanek in
dotnet/msbuild#13503
* Strengthen reviewer skill: add step-back analysis dimensions by
@​JanProvaznik in dotnet/msbuild#13504
* Add 'Request Speedometer Perf Run' to VS experimental insertion build
policies by @​Copilot in dotnet/msbuild#13505
* Remove duplicate @ prefix from issueAuthor in GitOps by @​akoeplinger
in dotnet/msbuild#13492
* Improve review aw by @​JanKrivanek in
dotnet/msbuild#13510
* Migrates unit tests to use RoslynCodeTaskFactory to enable running
tests under .NET Core by @​jankratochvilcz in
dotnet/msbuild#13500
* Fix cross-AppDomain TaskItem modifier cache regression by
@​DustinCampbell in dotnet/msbuild#13493
* Discourage review agent from approving PRs by @​JanKrivanek in
dotnet/msbuild#13512
* Stop trying to deploy ValueTuple by @​rainersigwald in
dotnet/msbuild#13507
* Ad-hoc re-sign bootstrap dotnet on macOS to prevent SIGKILL by
@​jankratochvilcz in dotnet/msbuild#13513
* RoslynCodeTaskFactory: Log MSB3753 when task class does not implement
ITask by @​jankratochvilcz in
dotnet/msbuild#13517
* Update gh-aw (upon mcp policy changes) by @​JanKrivanek in
dotnet/msbuild#13526
* Eliminate XmlChildNodes allocations in GetXmlNodeInnerContents by
@​nareshjo in dotnet/msbuild#13509
* Fix telemetry allocation regression: per-engine collector ownership by
@​JanProvaznik in dotnet/msbuild#13516
* Migrate to xunit.v3 by @​Youssef1313 in
dotnet/msbuild#13482
* Fix stray brace in HandleBuildCancel trace string causing MSB1025 by
@​Copilot in dotnet/msbuild#13535
* Bumping to 10.0.4 runtime packages by @​MichalPavlik in
dotnet/msbuild#13533
* Remove early return in GetCanonicalForm, always call System.IO.Path by
@​OvesN in dotnet/msbuild#13532
* Do not overwrite GetCopyToOutputDirectoryItemsDependsOn, just add new…
by @​snechaev in dotnet/msbuild#13474
* Migrate GetReferenceAssemblyPaths task to TaskEnvironment API by
@​OvesN in dotnet/msbuild#13495
* Stabilize ToolTaskThatTimeoutAndRetry test by @​rainersigwald in
dotnet/msbuild#13489
* [automated] Merge branch 'vs18.6' => 'main' by @​github-actions[bot]
in dotnet/msbuild#13506
* Add extra test assertions around tests by @​Youssef1313 in
dotnet/msbuild#13536
* Add static eval for repo skills/agents via skill-validator by
@​JanKrivanek in dotnet/msbuild#13537
* Migrate SGen task to Task environment API by @​OvesN in
dotnet/msbuild#13457
* Fix TerminalLogger assert failure for metaproj files and cached
project eval ID by @​OvesN in
dotnet/msbuild#13480
* Filter out approving review from pr-reviewer agent by @​JanKrivanek in
dotnet/msbuild#13553
* Use a unique task name per invocation to tabilize
RoslynCodeTaskFactory_ReuseCompilation test by @​huulinhnguyen-dev in
dotnet/msbuild#13551
* Brief doc on feedback/logging/data systems by @​rainersigwald in
dotnet/msbuild#13554
* Localized file check-in by OneLocBuild Task: Build definition ID 9434:
Build ID 13881982 by @​dotnet-bot in
dotnet/msbuild#13437
* Stage 3: Forward BuildProjectFile* callbacks from OOP TaskHost to
worker node by @​JanProvaznik in
dotnet/msbuild#13350
* Enable TaskHost Callbacks by default by @​JanProvaznik in
dotnet/msbuild#13579
* Remove unactionable info from reviewer agent by @​JanKrivanek in
dotnet/msbuild#13578
* Enlighten RequiresFramework35SP1Assembly task for multithreaded mode
by @​jankratochvilcz in dotnet/msbuild#13575
* Make SdkResolver-provided environment variables take precedence over
ambient environment by @​Copilot in
dotnet/msbuild#12655
* Add dotnet/skills marketplace and enable plugins by @​Evangelink in
dotnet/msbuild#13582
* The skills/agents check filters-in only touched files by @​JanKrivanek
in dotnet/msbuild#13586
* Fix skill-validation workflow failing when agents directory is deleted
by @​JeremyKuhne in dotnet/msbuild#13592
 ... (truncated)

Commits viewable in [compare
view](dotnet/msbuild@v18.6.3...v18.7.1).
</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants