Add static eval for repo skills/agents via skill-validator#13537
Conversation
There was a problem hiding this comment.
Pull request overview
Adds automated static validation for repository Copilot skills and agents using skill-validator, and updates existing repo references/content to satisfy validator expectations.
Changes:
- Added a PR/push-triggered workflow to run
skill-validatorand upload results as an artifact. - Added a
workflow_runfollow-up workflow to post (or update) a PR comment with summarized validator results. - Added/updated skills and agent content (including a shared workflow doc reference fix) to align with the validator.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/skill-validation.yml | Runs skill-validator on skills/agents, summarizes results, uploads artifact for downstream comment workflow. |
| .github/workflows/skill-validation-comment.yml | Downloads validator artifact and posts/updates a PR comment summarizing findings. |
| .github/workflows/shared/review-shared.md | Updates agent filename reference to .agent.md. |
| .github/skills/optimizing-msbuild-performance/SKILL.md | New performance-focused skill content. |
| .github/skills/maintaining-binary-log-compatibility/SKILL.md | New binlog compatibility skill content. |
| .github/skills/integrating-sdk-and-msbuild/SKILL.md | New SDK↔MSBuild integration skill content. |
| .github/skills/authoring-errors-and-warnings/SKILL.md | New diagnostics authoring skill content. |
| .github/skills/assessing-breaking-changes/SKILL.md | New breaking-change assessment skill content. |
| .github/skills/changewaves/SKILL.md | Fixes skill link target to the new assessing-breaking-changes skill. |
| .github/agents/expert-reviewer.agent.md | Adds/renames the expert reviewer agent definition and guidance. |
Copilot's findings
- Files reviewed: 4/10 changed files
- Comments generated: 3
There was a problem hiding this comment.
Review Summary
Renames (items 1-6): Clean. All old directory names have been grepped across the repo — no stale references remain in .md, .yml, or other files. The changewaves/SKILL.md cross-reference fix (item 7) and review-shared.md agent path update (item 8) are correct.
Security model: The workflow_run split between skill-validation.yml (read-only, runs PR code) and skill-validation-comment.yml (write permissions, never checks out PR code) correctly follows the safe fork-PR pattern. ✅
Findings
| Severity | Finding |
|---|---|
Hardcoded counts in skill-validation.yml:131-134 — skill-count (13), agent-count (1), total (14) are static and will silently go stale. Should be computed dynamically. |
|
| i️ | Unpinned nightly binary — skill-validator-nightly tag is mutable; a bad upstream release could break CI. Consider pinning or adding checksum verification. |
| i️ | No graceful handling if artifact upload was skipped — the comment workflow will fail hard if the validation workflow was cancelled before uploading. |
| i️ | Success summary is sparse — on a clean run the step summary just says "All checks passed" with no scope detail. |
No breaking changes, no security issues, no stale references found. The hardcoded counts are the main item worth addressing before merge.
Note
🔒 Integrity filter blocked 1 item
The following item were blocked because they don't meet the GitHub integrity level.
- #13537
pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
To allow these resources, lower min-integrity in your GitHub frontmatter:
tools:
github:
min-integrity: approved # merged | approved | unapproved | noneGenerated by Expert Code Review (on open) for issue #13537 · ● 2.5M
… handling, success summary
#13537 followup ### Context The PR above introduced skills/agents files checking within PRs touching skills/agents But it checked all the files - possibly repeating same and unactionable findings ### Changes Made We now filter only for the touched skills and agent files
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Description of Change Adopting skills-validator This is inspired and sourced from similar integrations: * github/awesome-copilot#1195 * dotnet/runtime#126814 * dotnet/msbuild#13537 One important difference is that in this case we run as well in the LLM judging mode cc: @PureWeen - for awareness and for help with iterating on this :)
Updated [Microsoft.Build](https://github.com/dotnet/msbuild) from 18.6.3 to 18.7.1. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Build's releases](https://github.com/dotnet/msbuild/releases)._ ## 18.7.1 ## What's Changed * Fix TraceEngine file contention deadlock in multithreaded mode by @JanProvaznik in dotnet/msbuild#13446 * Remove duplicate test cases in MultithreadableTaskAnalyzer by @Youssef1313 in dotnet/msbuild#13483 * Ensure ThreadSafeTaskAnalyzer.Tests is considered as a unit test project by @Youssef1313 in dotnet/msbuild#13481 * Fix MSBuildTask0002 analyzer warnings in already-migrated tasks by @JanProvaznik in dotnet/msbuild#13466 * Fix race conditions in task host path resolution by @AR-May in dotnet/msbuild#13485 * Migrate ToolTask and Al task to TaskEnvironment API by @OvesN in dotnet/msbuild#13423 * Bump main to 18.7, add vs18.6 to merge flow by @MichalPavlik in dotnet/msbuild#13472 * Avoid allocations in GetHashCode implementations by @DustinCampbell in dotnet/msbuild#13475 * Add PATs rotation to agentic workflow(s) by @JanKrivanek in dotnet/msbuild#13496 * Fix ASP.NET WebSite projects to copy netstandard.dll facade when required by @JanProvaznik in dotnet/msbuild#13058 * Migrate AspNetCompiler to TaskEnvironment API by @OvesN in dotnet/msbuild#13424 * Add review workflow by @JanKrivanek in dotnet/msbuild#13503 * Strengthen reviewer skill: add step-back analysis dimensions by @JanProvaznik in dotnet/msbuild#13504 * Add 'Request Speedometer Perf Run' to VS experimental insertion build policies by @Copilot in dotnet/msbuild#13505 * Remove duplicate @ prefix from issueAuthor in GitOps by @akoeplinger in dotnet/msbuild#13492 * Improve review aw by @JanKrivanek in dotnet/msbuild#13510 * Migrates unit tests to use RoslynCodeTaskFactory to enable running tests under .NET Core by @jankratochvilcz in dotnet/msbuild#13500 * Fix cross-AppDomain TaskItem modifier cache regression by @DustinCampbell in dotnet/msbuild#13493 * Discourage review agent from approving PRs by @JanKrivanek in dotnet/msbuild#13512 * Stop trying to deploy ValueTuple by @rainersigwald in dotnet/msbuild#13507 * Ad-hoc re-sign bootstrap dotnet on macOS to prevent SIGKILL by @jankratochvilcz in dotnet/msbuild#13513 * RoslynCodeTaskFactory: Log MSB3753 when task class does not implement ITask by @jankratochvilcz in dotnet/msbuild#13517 * Update gh-aw (upon mcp policy changes) by @JanKrivanek in dotnet/msbuild#13526 * Eliminate XmlChildNodes allocations in GetXmlNodeInnerContents by @nareshjo in dotnet/msbuild#13509 * Fix telemetry allocation regression: per-engine collector ownership by @JanProvaznik in dotnet/msbuild#13516 * Migrate to xunit.v3 by @Youssef1313 in dotnet/msbuild#13482 * Fix stray brace in HandleBuildCancel trace string causing MSB1025 by @Copilot in dotnet/msbuild#13535 * Bumping to 10.0.4 runtime packages by @MichalPavlik in dotnet/msbuild#13533 * Remove early return in GetCanonicalForm, always call System.IO.Path by @OvesN in dotnet/msbuild#13532 * Do not overwrite GetCopyToOutputDirectoryItemsDependsOn, just add new… by @snechaev in dotnet/msbuild#13474 * Migrate GetReferenceAssemblyPaths task to TaskEnvironment API by @OvesN in dotnet/msbuild#13495 * Stabilize ToolTaskThatTimeoutAndRetry test by @rainersigwald in dotnet/msbuild#13489 * [automated] Merge branch 'vs18.6' => 'main' by @github-actions[bot] in dotnet/msbuild#13506 * Add extra test assertions around tests by @Youssef1313 in dotnet/msbuild#13536 * Add static eval for repo skills/agents via skill-validator by @JanKrivanek in dotnet/msbuild#13537 * Migrate SGen task to Task environment API by @OvesN in dotnet/msbuild#13457 * Fix TerminalLogger assert failure for metaproj files and cached project eval ID by @OvesN in dotnet/msbuild#13480 * Filter out approving review from pr-reviewer agent by @JanKrivanek in dotnet/msbuild#13553 * Use a unique task name per invocation to tabilize RoslynCodeTaskFactory_ReuseCompilation test by @huulinhnguyen-dev in dotnet/msbuild#13551 * Brief doc on feedback/logging/data systems by @rainersigwald in dotnet/msbuild#13554 * Localized file check-in by OneLocBuild Task: Build definition ID 9434: Build ID 13881982 by @dotnet-bot in dotnet/msbuild#13437 * Stage 3: Forward BuildProjectFile* callbacks from OOP TaskHost to worker node by @JanProvaznik in dotnet/msbuild#13350 * Enable TaskHost Callbacks by default by @JanProvaznik in dotnet/msbuild#13579 * Remove unactionable info from reviewer agent by @JanKrivanek in dotnet/msbuild#13578 * Enlighten RequiresFramework35SP1Assembly task for multithreaded mode by @jankratochvilcz in dotnet/msbuild#13575 * Make SdkResolver-provided environment variables take precedence over ambient environment by @Copilot in dotnet/msbuild#12655 * Add dotnet/skills marketplace and enable plugins by @Evangelink in dotnet/msbuild#13582 * The skills/agents check filters-in only touched files by @JanKrivanek in dotnet/msbuild#13586 * Fix skill-validation workflow failing when agents directory is deleted by @JeremyKuhne in dotnet/msbuild#13592 ... (truncated) Commits viewable in [compare view](dotnet/msbuild@v18.6.3...v18.7.1). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Context
Let's use the skill-validator to check for basic problems of our skills and agents
This is inspired and sourced from similar integrations:
Changes Made
Testing