feat(scripts): add folder-consistency check and standardize WARN outp… by mariekekortsmit · Pull Request #1350 · microsoft/hve-core

mariekekortsmit · 2026-04-13T14:51:55Z

Description

Add a collection-id to folder name consistency check in Validate-Collections.ps1 that warns when a manifest item's folder doesn't match the collection id. Exempts shared/, hve-core/ folders and the hve-core-all collection. Also standardizes all Write-Warning calls to the Write-Host WARN pattern used throughout the script and normalizes WARN prefix spacing.

Add collection-id to folder name consistency check with exemptions for shared, hve-core, and hve-core-all
Replace Write-Warning calls with Write-Host WARN pattern for consistent output
Normalize WARN prefix spacing across all advisory messages
Add folder-consistency Pester tests with positive and negative Write-Host mock assertions

Related Issue(s)

Closes #1209

Type of Change

Select all that apply:

Code & Documentation:

Bug fix (non-breaking change fixing an issue)
New feature (non-breaking change adding functionality)
Breaking change (fix or feature causing existing functionality to change)
Documentation update

Infrastructure & Configuration:

AI Artifacts:

Reviewed contribution with prompt-builder agent and addressed all feedback
Copilot instructions (.github/instructions/*.instructions.md)
Copilot prompt (.github/prompts/*.prompt.md)
Copilot agent (.github/agents/*.agent.md)
Copilot skill (.github/skills/*/SKILL.md)

Other:

Script/automation (.ps1, .sh, .py)
Other (please describe):

Testing

All 45 Pester tests pass: npm run test:ps -- -TestPath "scripts/tests/collections/Validate-Collections.Tests.ps1"
6 new folder-consistency tests cover: matching folder (no WARN), mismatched folder (WARN emitted), hve-core/ exemption, shared/ exemption, hve-core-all skip, and duplicate WARN output assertion
Tests use Mock Write-Host {} with Should -Invoke / Should -Not -Invoke and ParameterFilter matching WARN collection anchor

Checklist

Required Checks

Documentation is updated (if applicable)
Files follow existing naming conventions
Changes are backwards compatible (if applicable)
Tests added for new functionality (if applicable)

Required Automated Checks

The following validation commands must pass before merging:

Markdown linting: npm run lint:md
Spell checking: npm run spell-check
Frontmatter validation: npm run lint:frontmatter
Skill structure validation: npm run validate:skills
Link validation: npm run lint:md-links
PowerShell analysis: npm run lint:ps
Plugin freshness: npm run plugin:generate
Docusaurus tests: npm run docs:test

Security Considerations

This PR does not contain any sensitive or NDA information
Any new dependencies have been reviewed for security issues
Security-related scripts follow the principle of least privilege

Additional Notes

The folder-consistency check is advisory only (WARN, not FAIL) — it does not block validation
Write-Warning was the outlier pattern; Write-Host WARN is the established convention in this script per maintainer direction

…ut in collection validation - add collection-id to folder name consistency check with exemptions for shared, hve-core, and hve-core-all - replace Write-Warning calls with Write-Host WARN pattern for consistent output - normalize WARN prefix spacing across all advisory messages - add folder-consistency Pester tests with positive and negative Write-Host mock assertions ✨ - Generated by Copilot

codecov-commenter · 2026-04-13T14:54:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.65%. Comparing base (a5c3837) to head (16e0c3a).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1350      +/-   ##
==========================================
- Coverage   87.92%   87.65%   -0.27%     
==========================================
  Files          62       61       -1     
  Lines        9593     9335     -258     
==========================================
- Hits         8435     8183     -252     
+ Misses       1158     1152       -6

Flag	Coverage Δ
pester	`85.24% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
scripts/collections/Validate-Collections.ps1	`93.14% <100.00%> (+0.41%)`	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

katriendg

Thanks @mariekekortsmit for your contribution!

As I reviewed the current collections and what we are trying to achieve, we want to allow for some of the intentional cross-collection bundling. We want to catch folder names not matching collection names, but allow cross-bundling.

Could you update the check against all known collection IDs derived from the *.collection.yml filenames in collections folder. This catches real stale folders (like the original code-review/ → coding-standards/ drift from #1208) while allowing intentional cross-collection bundling.

Suggested fix:

Build a $knownCollectionIds set from all manifest filenames before the per-file loop.
Change the condition from $folderName -ne $id to -not $knownCollectionIds.ContainsKey($folderName).
Update the warning message to: "folder does not match any known collection ID".

You will probably just end up with one warning about a skill for playwright which we are aware of.
Thanks!

mariekekortsmit · 2026-04-14T12:32:59Z

Thanks @mariekekortsmit for your contribution!

As I reviewed the current collections and what we are trying to achieve, we want to allow for some of the intentional cross-collection bundling. We want to catch folder names not matching collection names, but allow cross-bundling.

Could you update the check against all known collection IDs derived from the *.collection.yml filenames in collections folder. This catches real stale folders (like the original code-review/ → coding-standards/ drift from #1208) while allowing intentional cross-collection bundling.

Suggested fix:

Build a $knownCollectionIds set from all manifest filenames before the per-file loop.

Change the condition from $folderName -ne $id to -not $knownCollectionIds.ContainsKey($folderName).

Update the warning message to: "folder does not match any known collection ID".

You will probably just end up with one warning about a skill for playwright which we are aware of. Thanks!

So that means the relation should be bidirectional? All items under a specific prefix folder (e.g. .github/skills/experimental) should be inside the related collection.yaml, AND the collection should contain only items from the associated folder? E.g. in the playwright case, it is a skill inside skills/experimental/vscode-playwright folder, but it does not show in the experimental.collection.yaml, and that should issue a warning.

I only understood a one way relation, that explains our difference.

katriendg · 2026-04-14T13:19:06Z

@mariekekortsmit The warning with the playwright is correct because it is missing in a collection experimental.

The issue with the current approach is it will also warn for an item from project-planning which is included in security collection, which is intentional. So in future, if a new folder my-collection is created, and this item is not found in a matching my-collection.collection.yml, that's when we want to warn.

Maybe in future we will want to warn about cross-bundling, but the current version contains several which we accept and don't want to warn about.

Hope that makes sense?

mariekekortsmit · 2026-04-15T10:18:06Z

@mariekekortsmit The warning with the playwright is correct because it is missing in a collection experimental.

The issue with the current approach is it will also warn for an item from project-planning which is included in security collection, which is intentional. So in future, if a new folder my-collection is created, and this item is not found in a matching my-collection.collection.yml, that's when we want to warn.

Maybe in future we will want to warn about cross-bundling, but the current version contains several which we accept and don't want to warn about.

Hope that makes sense?

Maybe I'm confused because of the word "cross-bundling" and what that means exactly so
So the only case you want warnings for is if a new folder my-collection is created (so that means a folder underneath .github/agents/, .github/prompts/, .github/instructions/ and/or .github/skills/), and this item is not found in a matching my-collection.collection.yml, that's when we want to warn.

And with cross-bundling you mean, there is no check needed to verify that every item named in my-collection.collection.yaml indeed also appears in a folder that looks like .github/*/my-collection.

In the issue, one of the acceptance criteria is:
"The hve-core-all collection is exempt (it intentionally bundles items from all collections)"
But my current understanding based on your comments in this PR leads to all collections being exempt from that check, not only hve-core-all. Is that correct?

If my understanding still does not match the idea, maybe we can have a quick call to figure out where the misunderstanding is.

katriendg · 2026-04-15T13:34:41Z

@mariekekortsmit The warning with the playwright is correct because it is missing in a collection experimental.
The issue with the current approach is it will also warn for an item from project-planning which is included in security collection, which is intentional. So in future, if a new folder my-collection is created, and this item is not found in a matching my-collection.collection.yml, that's when we want to warn.
Maybe in future we will want to warn about cross-bundling, but the current version contains several which we accept and don't want to warn about.
Hope that makes sense?

Maybe I'm confused because of the word "cross-bundling" and what that means exactly so So the only case you want warnings for is if a new folder my-collection is created (so that means a folder underneath .github/agents/, .github/prompts/, .github/instructions/ and/or .github/skills/), and this item is not found in a matching my-collection.collection.yml, that's when we want to warn.

And with cross-bundling you mean, there is no check needed to verify that every item named in my-collection.collection.yaml indeed also appears in a folder that looks like .github/*/my-collection.

In the issue, one of the acceptance criteria is: "The hve-core-all collection is exempt (it intentionally bundles items from all collections)" But my current understanding based on your comments in this PR leads to all collections being exempt from that check, not only hve-core-all. Is that correct?

If my understanding still does not match the idea, maybe we can have a quick call to figure out where the misunderstanding is.

I apologize for the confusion, the initial issue had the aim to also warn about cross-bundling. With cross-bundling we mean that an item from a collection project-planning is also referenced in security or vice-versa. Initially this was not the plan, which is what the issue describes as only hve-core and shared being allowed in cross-collection bundling.
But since we have others bundled, in the end I will update the acceptance criteria to match our current reality. My intention is for the future to fine tune this more towards our initial goal, but today we especially want to prevent a new folder/artifacts being created and not added to any collection. That's what we need to warn on.

…-consistency validation Replace per-collection id comparison with a knownCollectionIds lookup built from all *.collection.yml filenames. This allows intentional cross-collection bundling (e.g. project-planning items in security collection) while still warning when an item's folder matches no known collection at all. Remove the hardcoded hve-core exemption since hve-core.collection.yml exists as a known collection ID. Retain the shared exemption as there is no matching collection manifest for that folder. Update test for hve-core/ folder to register hve-core as a known collection by creating a minimal hve-core.collection.yml + companion .md inline, matching real-world behavior.

katriendg · 2026-04-23T14:23:44Z

Hey @mariekekortsmit, we updated your branch with some changes in the collection checks for orphaned items. No action required from you, as we wanted to get a number of things merged in before next release. Thanks for your contribution.

bindsi

Thanks for your contribution. LGTM!

## Pre-Release 3.3.101 ### ✨ Features - add removed maturity tier and retire owasp-docker (#1444) - add evaluation dataset creator (#1279) - align RAI planner with guide, remove scoring, improve UX (#1287) - add PSGallery staleness check and BOM cleanup (#1379) - ISA-95 network planner agent (#1177) - auto-generate collection.md with maturity filtering (#1316) - add folder-consistency check and standardize WARN outp… (#1350) - add synth-data-generate prompt to data-science collection (#1419) - add canonical deck workflow and customer-card rendering for design thinking (#1413) - add Figma MCP integration for DT artifact export (#1222) - introduce `owasp-docker` (#1245) - replace hve-core-specific references with portable discovery-based language (#1335) - introduce `owasp-cicd` (#1246) - add secure-by-design knowledge skill (#1223) - introduce `owasp-infrastructure` (#1244) - introduce `owasp-mcp` (#1207) - add OutputPath parameter to Invoke-LinkLanguageCheck.ps1 (#1229) - add -OutputPath parameter to Validate-SkillStructure.ps1 (#1225) - add maintainer-only skip-review label guard (#1293) - add extension collections overview and integrate into getting started flow (#950) - add agentic workflows for automated issue triage, implementation, PR review, dependency review, and doc-staleness detection (#1219) - consolidate package-lock.json version sync into Update-VersionFiles.ps1 (#1240) - add standards code review agent and full review orchestrator (#1174) - standardize pytest-mock as Python mocking framework (#1170) - add Jira backlog workflows and Jira/GitLab skills (#978) - add centralized version bump script and supply-chain attestation (#1183) ### 🐛 Bug Fixes - pin PowerShell-Yaml to 0.4.7 across all install sites (#1378) - close fork-PR/workflow-file-PR secret-strip gap and normalize upload-artifact version (#1421) - replace stream-based lookahead with array indexing in list-changed-files.sh (#1376) - centralize ISO 8601 timestamp regex in CIHelpers (#1343) - update stale documentation date in release-process.md (#1363) - pin basic-ftp to 5.3.0 to resolve GHSA-rp42-5vxx-qpwr (#1374) - add bot filter to dependency PR review workflow (#1362) - resolve pip-audit findings in powerpoint, gitlab, and jira skill lock files (#1360) - standardize Timestamp JSON key casing across all lint result files (#1314) - add synchronize trigger to PR Review workflow (#1323) - standardize timestamp in Validate-SkillStructure.ps1 to use Get-StandardTimestamp (#1280) - add parallel subagent dispatch and structured JSON contracts to code-review-full (#1304) - standardize timestamp in SecurityHelpers.psm1 to use Get-StandardTimestamp (#1284) - standardize timestamps in Test-DependencyPinning.ps1 and SecurityClasses.psm1 (#1282) - derive collection artifact counts from YAML at build time (#1275) - standardize timestamp in FrontmatterValidation.psm1 to use Get-StandardTimestamp (#1285) - standardize timestamp in Markdown-Link-Check.ps1 to use Get-StandardTimestamp (#1283) - escape hyphens in Mermaid diagram on Collections page (#1262) - add summary timestamp to PSScriptAnalyzer output (#1211) - fix plugin compatibility and robustness for coding-standards code review agents (#1289) - standardize timestamp in Test-CopyrightHeaders.ps1 to use Get-StandardTimestamp (#1278) - standardize timestamp in Invoke-YamlLint.ps1 to use Get-StandardTimestamp (#1270) - standardize timestamp in Invoke-LinkLanguageCheck.ps1 to use Get-StandardTimestamp (#1264) - fix dependency-review path filters and sparse-checkout cone mode (#1259) - replace invalid bare tool names with official tool identifiers (#1198) - fix broken links and remove orphaned reference in code review docs (#1257) - exclude Python env dirs from skill validation warnings (#1255) - pin happy-dom and serialize-javascript to resolve Dependabot vulnerabilities (#1253) - remove Mermaid diagram and add missing collection cards (#1247) - disable MCP servers by default to prevent token limit errors (#1144) - sync package-lock.json after pre-release version bump (#1236) - separate mermaid node declarations and add dynamic diagram generation with tests (#1215) - replace anchor links in meeting-analyst with bold text references (#1201) - remove recursive symlinks in jira and gitlab skill directories (#1233) - validate-installation scripts now check .github/skills directory (#1010) (#1206) - resolve npm audit vulnerabilities via dependency overrides (#1200) - add post-release triggers to scorecard workflow (#1186) - add missing .md extensions to relative links in agent documentation (#1180) ### 📚 Documentation - broaden Security Review description beyond OWASP (#1385) - document maintainer advisory mode and skip-review label guard (#1386) - document ExcludePaths/OutputPath for Invoke-LinkLanguageCheck (#1383) - CLI getting-started: clarify plugin install commands as alternatives (-all vs base) (#1251) ### ♻️ Refactoring - align agent and prompt folder names to collection identifier (#1210) ### 🔧 Maintenance - pin PSScriptAnalyzer to 1.25.0 and sync stale workflow version comments (#1389) - bump lxml from 6.0.2 to 6.1.0 in /.github/skills/experimental/powerpoint (#1424) - bump @vscode/vsce from 3.7.1 to 3.9.1 in the npm-dependencies group (#1390) - bump the github-actions group across 1 directory with 7 updates (#1391) - bump follow-redirects from 1.15.11 to 1.16.0 in /docs/docusaurus (#1356) - upgrade Node.js from 20 to 24 and bump cspell to v10 (#1353) - bump basic-ftp from 5.2.0 to 5.2.1 (#1324) - update github/gh-aw-actions requirement to 536ea1bad8c6715d098a9dc1afea8d403733acfe in the github-actions group across 1 directory (#1298) - update security instruction attributions and compliance (#1294) - bump the npm-dependencies group with 2 updates (#1297) - pre-release 3.3.41 (#1252) - streamline RAI Planner phase structure and documentation (#1273) - bump happy-dom from 20.8.8 to 20.8.9 in /docs/docusaurus (#1237) - pre-release 3.3.27 (#1191) - bump pygments from 2.19.2 to 2.20.0 in /.github/skills/gitlab/gitlab (#1234) - bump path-to-regexp from 0.1.12 to 0.1.13 in /docs/docusaurus (#1226) - bump the github-actions group with 4 updates (#1231) - add missing folders and alphabetize location lists (#1193) - bump brace-expansion (#1224) - bump handlebars from 4.7.8 to 4.7.9 in /docs/docusaurus (#1217) - bump brace-expansion from 5.0.3 to 5.0.5 in /docs/docusaurus (#1213) - pre-release 3.3.10 (#1187) - bump markdownlint-cli2 from 0.21.0 to 0.22.0 in the npm-dependencies group (#1175) - bump the github-actions group with 3 updates (#1176) - pre-release 3.3.1 (#1165) --- *Managed automatically by pre-release workflow.* Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

mariekekortsmit requested a review from a team as a code owner April 13, 2026 14:51

Merge branch 'main' into feat/issue-1209-collection-folder-consistency

ef79d01

katriendg reviewed Apr 14, 2026

View reviewed changes

WilliamBerryiii and others added 3 commits April 22, 2026 23:17

Merge branch 'main' into feat/issue-1209-collection-folder-consistency

5f8ddfe

Merge branch 'main' into feat/issue-1209-collection-folder-consistency

0a5d397

katriendg approved these changes Apr 23, 2026

View reviewed changes

bindsi approved these changes Apr 23, 2026

View reviewed changes

bindsi merged commit 410de7d into microsoft:main Apr 23, 2026
50 checks passed

hve-core-release-please Bot mentioned this pull request Apr 23, 2026

chore(main): release hve-core 3.4.0 #1184

Open

github-actions Bot mentioned this pull request Apr 23, 2026

[aw] No-Op Runs #1265

Open

hve-core-release-please Bot mentioned this pull request Apr 24, 2026

chore(main): pre-release 3.3.101 #1274

Merged

hve-core-release-please Bot mentioned this pull request Apr 27, 2026

chore(main): pre-release 3.3.111 #1476

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(scripts): add folder-consistency check and standardize WARN outp…#1350

feat(scripts): add folder-consistency check and standardize WARN outp…#1350
bindsi merged 5 commits intomicrosoft:mainfrom
mariekekortsmit:feat/issue-1209-collection-folder-consistency

mariekekortsmit commented Apr 13, 2026

Uh oh!

codecov-commenter commented Apr 13, 2026 •

edited

Loading

Uh oh!

katriendg left a comment

Uh oh!

mariekekortsmit commented Apr 14, 2026

Uh oh!

katriendg commented Apr 14, 2026

Uh oh!

mariekekortsmit commented Apr 15, 2026 •

edited

Loading

Uh oh!

katriendg commented Apr 15, 2026

Uh oh!

katriendg commented Apr 23, 2026

Uh oh!

bindsi left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mariekekortsmit commented Apr 13, 2026

Description

Related Issue(s)

Type of Change

Testing

Checklist

Required Checks

Required Automated Checks

Security Considerations

Additional Notes

Uh oh!

codecov-commenter commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

katriendg left a comment

Choose a reason for hiding this comment

Uh oh!

mariekekortsmit commented Apr 14, 2026

Uh oh!

katriendg commented Apr 14, 2026

Uh oh!

mariekekortsmit commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

katriendg commented Apr 15, 2026

Uh oh!

katriendg commented Apr 23, 2026

Uh oh!

bindsi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Apr 13, 2026 •

edited

Loading

mariekekortsmit commented Apr 15, 2026 •

edited

Loading