Skip to content

Add vmr-codeflow-status Copilot skill#124109

Merged
lewing merged 19 commits intomainfrom
lewing/vmr-codeflow-skill
Feb 7, 2026
Merged

Add vmr-codeflow-status Copilot skill#124109
lewing merged 19 commits intomainfrom
lewing/vmr-codeflow-skill

Conversation

@lewing
Copy link
Member

@lewing lewing commented Feb 6, 2026 •
edited
Loading

Summary

Adds a new Copilot Agent Skill (.github/skills/vmr-codeflow-status/) that automates analysis of VMR codeflow PR health for dotnet repositories. Handles both backflow (VMR → product repos) and forward flow (product repos → VMR).

What it does

PR Analysis Mode:

  1. Parses PR metadata — Extracts VMR commit, subscription ID, build info from the PR body
  2. Validates snapshot — Cross-references PR body commit against branch "Backflow from" commit messages to detect stale metadata
  3. Checks freshness — Compares PR's snapshot against the current source branch HEAD, showing how many commits behind and which repos have updates
  4. Shows pending forward flow — For behind backflow PRs, finds open forward flow PRs that would close part of the freshness gap
  5. Detects staleness & conflicts — Finds Maestro "codeflow cannot continue" warnings and "Conflict detected" messages (with conflicting files and resolve commands)
  6. Analyzes PR commits — Categorizes commits as auto-updates vs manual (which would be lost on close/force-trigger)
  7. Traces fixes (optional) — Checks if a specific fix PR has flowed through VMR into the codeflow PR
  8. Recommends actions — Suggests force trigger, close/reopen, merge as-is, resolve conflicts, or wait, with ready-to-use darc commands

Missing Backflow Mode:

  • Discovers all branches with backflow subscriptions by examining recently merged PRs
  • Compares VMR branch HEAD against the last merged backflow commit
  • Flags branches where the VMR has moved but no new backflow PR exists
  • Warns when Maestro may be stuck (>6 hours since last merge with VMR ahead)

Example Prompts

Motivation

Investigating stale codeflow PRs is a common and tedious task — checking VMR freshness, finding staleness warnings, tracing whether specific fixes have flowed through. This skill automates the entire workflow and provides actionable darc commands.

Tested against 10+ codeflow PRs across dotnet/sdk, dotnet/runtime, dotnet/roslyn, dotnet/aspnetcore, and dotnet/dotnet on multiple branches (main, preview1, release/10.0.x).

Files

  • SKILL.md — Skill definition with frontmatter (matching existing skills in this repo)
  • scripts/Get-CodeflowStatus.ps1 — Main analysis script (~750 lines)
  • references/vmr-codeflow-reference.md — Reference docs (VMR concepts, darc CLI, API endpoints)

@lewing
Add vmr-codeflow-status Copilot skill
a86f37c 
Adds a Copilot Agent Skill that analyzes VMR codeflow PR health:
- Parses PR metadata (subscription ID, VMR commit, build info)
- Checks VMR freshness (compares PR snapshot vs branch HEAD)
- Detects Maestro staleness warnings
- Analyzes PR commits (auto-updates vs manual)
- Traces specific fixes through VMR pipeline (-TraceFix)
- Provides actionable recommendations with darc commands
Copilot AI review requested due to automatic review settings February 6, 2026 22:15
@github-actions github-actions bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Feb 6, 2026
@lewing lewing marked this pull request as draft February 6, 2026 22:16
Copilot AI reviewed Feb 6, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Copilot Agent Skill (vmr-codeflow-status) to automate health analysis of VMR backflow/codeflow PRs by querying GitHub PR metadata, VMR branch freshness, Maestro staleness warnings, and (optionally) tracing whether a specific fix has flowed through VMR.

Changes:

  • Introduces a PowerShell script to analyze codeflow PR freshness, staleness, commit makeup, and optional fix tracing.
  • Adds skill documentation (SKILL.md) and a VMR/codeflow reference doc explaining concepts, metadata formats, and darc commands.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.

File Description
.github/skills/vmr-codeflow-status/scripts/Get-CodeflowStatus.ps1 Implements the PR/VMR analysis logic and prints status + recommendations.
.github/skills/vmr-codeflow-status/references/vmr-codeflow-reference.md Provides background/reference material for VMR codeflow concepts and tooling.
.github/skills/vmr-codeflow-status/SKILL.md Defines the skill and documents usage parameters and interpretation.

lewing added 2 commits February 6, 2026 16:28
@lewing
Fix bugs and simplify API calls
66f722b 
Bug fixes:
- Use full SHAs in compare API calls (Get-ShortSha for display only)
- Fix Write-Status 3rd arg bug that would throw on color parse
- Remove dead code
- Handle behind/diverged status in VMR compare, not just ahead
- Remove extra frontmatter fields to match existing skills

Simplifications:
- Replace 3 separate REST calls with single gh pr view --json
- Use raw content header for source-manifest.json (no Base64)
@lewing
Fix remaining review issues
a2b6556 
- Use merged_at instead of merged boolean for fix PR detection
- URL-encode VMR branch names with / in API paths
- Fix Accept header: application/vnd.github.raw (not raw+json)
- Fix contradictory recommendation message to use compare status
- Initialize behindBy/compareStatus at script scope
Copilot AI review requested due to automatic review settings February 6, 2026 22:37
Copilot AI reviewed Feb 6, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

@lewing
Remove dead code and join output before ConvertFrom-Json
fb581d2 
- Remove unused \ and \ variables
- Join gh api / gh pr view output before ConvertFrom-Json for robustness
Copilot AI review requested due to automatic review settings February 6, 2026 22:47
@lewing
Add forward flow support
3db9577 
- Auto-detect flow direction from PR title
- Backflow: 'Source code updates from dotnet/dotnet' (VMR -> product repo)
- Forward flow: 'Source code updates from dotnet/<repo>' (repo -> VMR)
- Adjust freshness comparison to use correct source repo
- Use flow-aware labels (VMR Commit vs Source Commit, etc.)
@lewing lewing force-pushed the lewing/vmr-codeflow-skill branch from 7f18b9c to 3db9577 Compare February 6, 2026 22:50
Copilot AI reviewed Feb 6, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Copilot AI review requested due to automatic review settings February 6, 2026 23:00
Copilot AI reviewed Feb 6, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Copilot AI review requested due to automatic review settings February 6, 2026 23:19
Copilot AI reviewed Feb 6, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

@lewing
Copy link
Member Author

lewing commented Feb 6, 2026 •
edited
Loading 

❯ what is the current status of codeflow for runtime

● skill(vmr-codeflow-status)

● Let me check the codeflow status for the runtime repository.

● Check runtime codeflow status
  $ & "C:\Users\lewing\.copilot\skills\vmr-codeflow-status\scripts\Get-CodeflowStatus.ps1" -Repository "dotnet/runtime"
  -CheckMissing
  └ 34 lines...

● Now let me get details on the two open backflow PRs.

● Check main branch codeflow PR
  $ & "C:\Users\lewing\.copilot\skills\vmr-codeflow-status\scripts\Get-CodeflowStatus.ps1" -Repository "dotnet/runtime"
  -PRNumber 124095
  └ 51 lines...

● Check preview1 codeflow PR
  $ & "C:\Users\lewing\.copilot\skills\vmr-codeflow-status\scripts\Get-CodeflowStatus.ps1" -Repository "dotnet/runtime"
  -PRNumber 124098
  └ 53 lines...
● Here's the current codeflow status for dotnet/runtime:

  -------------------------------------------------------------------------------------------------------------------

  Open Backflow PRs

  ┌───────────────────────┬─────────┬─────────────────┬───────────────────────┐
  │ Branch                │ PR      │ Status          │ VMR Behind            │
  ├───────────────────────┼─────────┼─────────────────┼───────────────────────┤
  │ main                  │ #124095 │ ⚠️ Needs update │ 33 commits behind VMR │
  ├───────────────────────┼─────────┼─────────────────┼───────────────────────┤
  │ release/11.0-preview1 │ #124098 │ ⚠️ Needs update │ 2 commits behind VMR  │
  └───────────────────────┴─────────┴─────────────────┴───────────────────────┘

  Missing Backflow PR

  ┌──────────────┬────────────────────────┬───────────────────────────────────────────────────────────────────────────
  ─────┐
  │ Branch       │ Issue                  │ Details
       │
  ├──────────────┼────────────────────────┼───────────────────────────────────────────────────────────────────────────
  ─────┤
  │ release/10.0 │ 🚨 Missing backflow PR │ VMR is 33 commits ahead, last PR merged ~22.6 hours ago — Maestro may be
  stuck │
  └──────────────┴────────────────────────┴───────────────────────────────────────────────────────────────────────────
  ─────┘

  Details

   - main (#124095): Created ~6 hours ago, no staleness warnings, no manual commits. Missing updates from 11 repos
  (aspnetcore, nuget, razor, runtime, efcore, roslyn, sdk, etc.). Maestro should auto-update, or you can trigger
  manually: darc trigger-subscriptions --id f7901f87-9f24-40d6-9bc1-564863937237
   - release/11.0-preview1 (#124098): Only 2 commits behind (missing fsharp updates), no staleness. 5 pending forward
  flow PRs into VMR. Should auto-update soon, or trigger: darc trigger-subscriptions --id
  96ab39a6-78f4-4265-9e1a-76875a9d2b5f
   - release/10.0: ⚠️ No open backflow PR exists despite VMR being 33 commits ahead. The last merged PR was ~22.6
  hours ago — this suggests Maestro may be stuck or the subscription needs investigation.

Copilot AI reviewed Feb 7, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Copilot AI review requested due to automatic review settings February 7, 2026 00:21
Copilot AI reviewed Feb 7, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

@lewing
Fix UTC time parsing, capture full resolve-conflict command, case-ins…
57559cd 
…ensitive commit SHA parsing, align SKILL.md commands
@lewing lewing marked this pull request as ready for review February 7, 2026 02:01
Copilot AI review requested due to automatic review settings February 7, 2026 02:01
Copilot AI reviewed Feb 7, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

@lewing
Copy link
Member Author

lewing commented Feb 7, 2026

@copilot Check the codeflow status of dotnet/sdk#52727 using the vmr-codeflow-status skill in this PR branch.

Copilot AI mentioned this pull request Feb 7, 2026
Closed
Copy link
Contributor

Copilot AI commented Feb 7, 2026

@lewing I've opened a new pull request, #124122, to work on those changes. Once the pull request is ready, I'll request review from you.

@lewing
Copy link
Member Author

lewing commented Feb 7, 2026

@copilot Run the script at .github/skills/vmr-codeflow-status/scripts/Get-CodeflowStatus.ps1 with -PRNumber 52727 -Repository 'dotnet/sdk' and post the output as a comment on this PR. Do not open a new PR or make any code changes.

Copilot AI mentioned this pull request Feb 7, 2026
Closed
Copy link
Contributor

Copilot AI commented Feb 7, 2026

@lewing I've opened a new pull request, #124123, to work on those changes. Once the pull request is ready, I'll request review from you.

@lewing
Copy link
Member Author

lewing commented Feb 7, 2026 •
edited
Loading

Observation: Coding agent doesn't support diagnostic/reporting skills

We tested invoking this skill via @copilot in PR comments. Both attempts resulted in the coding agent opening new PRs (#124122, #124123 — both closed) rather than running the script and posting the output.

The Agent Skills documentation says:

Agent Skills work with Copilot coding agent, the GitHub Copilot CLI and agent mode in Visual Studio Code Insiders.

And:

When Copilot chooses to use a skill, the SKILL.md file will be injected in the agent's context, giving the agent access to your instructions. It can then follow those instructions, and use any scripts or examples you may have included.

The coding agent likely does load the skill and may even run the script internally, but its only output mechanism is opening a PR with code changes. There's no way for it to "just report back" with diagnostic output in a comment.

This creates a gap for diagnostic/reporting skills (like this codeflow status tool) that are designed to analyze and report rather than modify code. These skills work great in the CLI and VS Code agent mode, but the coding agent can't use them effectively because:

  1. The output is informational, not a code change
  2. The appropriate response is a comment, not a PR
  3. Forcing results into a PR creates noise (unnecessary PRs that need to be closed)

Suggestion: The coding agent could support a "report" or "comment" action mode for skills that are diagnostic in nature, posting the script output as a PR/issue comment instead of always opening a new PR.

@copilot

jeffhandley
jeffhandley approved these changes Feb 7, 2026
View reviewed changes
Copy link
Member

@jeffhandley jeffhandley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bravo!

@lewing lewing merged commit 2d638dc into main Feb 7, 2026
17 checks passed
@lewing lewing deleted the lewing/vmr-codeflow-skill branch February 7, 2026 03:37
@lewing lewing added area-skills Agent Skills and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Feb 7, 2026
@lewing lewing mentioned this pull request Feb 7, 2026
Merged
@lewing
Copy link
Member Author

lewing commented Feb 7, 2026

Follow-up: #124129 promotes eng/Version.Details.xml as the primary VMR snapshot source (instead of commit message parsing). This correctly handles manual backflow operations like darc vmr backflow where the commit message doesn't follow the standard format.

@lewing lewing mentioned this pull request Feb 7, 2026
Merged
@dotnet-maestro dotnet-maestro bot mentioned this pull request Feb 8, 2026
Merged
lewing added a commit to lewing/runtime that referenced this pull request Feb 9, 2026
@lewing
Add vmr-codeflow-status Copilot skill (dotnet#124109)
20971b9 
## Summary

Adds a new Copilot Agent Skill (`.github/skills/vmr-codeflow-status/`)
that automates analysis of VMR codeflow PR health for dotnet
repositories. Handles both backflow (VMR → product repos) and forward
flow (product repos → VMR).

### What it does

**PR Analysis Mode:**
1. **Parses PR metadata** — Extracts VMR commit, subscription ID, build
info from the PR body
2. **Validates snapshot** — Cross-references PR body commit against
branch "Backflow from" commit messages to detect stale metadata
3. **Checks freshness** — Compares PR's snapshot against the current
source branch HEAD, showing how many commits behind and which repos have
updates
4. **Shows pending forward flow** — For behind backflow PRs, finds open
forward flow PRs that would close part of the freshness gap
5. **Detects staleness & conflicts** — Finds Maestro "codeflow cannot
continue" warnings and "Conflict detected" messages (with conflicting
files and resolve commands)
6. **Analyzes PR commits** — Categorizes commits as auto-updates vs
manual (which would be lost on close/force-trigger)
7. **Traces fixes** (optional) — Checks if a specific fix PR has flowed
through VMR into the codeflow PR
8. **Recommends actions** — Suggests force trigger, close/reopen, merge
as-is, resolve conflicts, or wait, with ready-to-use darc commands

**Missing Backflow Mode:**
- Discovers all branches with backflow subscriptions by examining
recently merged PRs
- Compares VMR branch HEAD against the last merged backflow commit
- Flags branches where the VMR has moved but no new backflow PR exists
- Warns when Maestro may be stuck (>6 hours since last merge with VMR
ahead)

### Example Prompts

- "check the codeflow status of dotnet/sdk#52727"
- "is dotnet#124098 up to date with the VMR?"
- "has the runtime revert in dotnet#123974 reached the codeflow
PR dotnet/sdk#52727?"
- "why is this codeflow PR blocked?"
- "are there any missing backflow PRs for dotnet/roslyn?"
- "report on the codeflow status for dotnet/aspnetcore"
- "what forward flow PRs are pending for the preview1 VMR branch?"

### Motivation

Investigating stale codeflow PRs is a common and tedious task — checking
VMR freshness, finding staleness warnings, tracing whether specific
fixes have flowed through. This skill automates the entire workflow and
provides actionable darc commands.

Tested against 10+ codeflow PRs across dotnet/sdk, dotnet/runtime,
dotnet/roslyn, dotnet/aspnetcore, and dotnet/dotnet on multiple branches
(main, preview1, release/10.0.x).

### Files

- `SKILL.md` — Skill definition with frontmatter (matching existing
skills in this repo)
- `scripts/Get-CodeflowStatus.ps1` — Main analysis script (~750 lines)
- `references/vmr-codeflow-reference.md` — Reference docs (VMR concepts,
darc CLI, API endpoints)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@jeffhandley jeffhandley jeffhandley approved these changes

Assignees

@lewing lewing

Labels

area-skills Agent Skills

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@lewing @jeffhandley