Skip to content

feat(collections): auto-generate collection.md with maturity filtering#1316

Merged
bindsi merged 15 commits intomainfrom
feat/collection-md-autogen-maturity-1256
Apr 23, 2026
Merged

feat(collections): auto-generate collection.md with maturity filtering#1316
bindsi merged 15 commits intomainfrom
feat/collection-md-autogen-maturity-1256

Conversation

@WilliamBerryiii
Copy link
Copy Markdown
Member

Description

Added auto-generation of collection.md artifact tables from YAML manifests with channel-based maturity filtering and sentinel marker support. The generation engine in Prepare-Extension.ps1 gained a -Channel parameter that controls which maturity levels appear in output, and New-CollectionReadme was rewritten to parse marker-delimited sections, filter items by maturity, and write back updated content preserving hand-written introductions and footers. Collection validation now detects mismatched or reversed markers as warnings for backward compatibility. The encoding for Set-ContentIfChanged was corrected to UTF-8 without BOM. Nineteen new Pester tests cover maturity filtering, template marker handling, and Split-CollectionMdByMarkers edge cases.

Related Issue(s)

Closes #1256

Type of Change

Select all that apply:

Code & Documentation:

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update

Infrastructure & Configuration:

  • GitHub Actions workflow
  • Linting configuration (markdown, PowerShell, etc.)
  • Security configuration
  • DevContainer configuration
  • Dependency update

AI Artifacts:

  • Reviewed contribution with prompt-builder agent and addressed all feedback
  • Copilot instructions (.github/instructions/*.instructions.md)
  • Copilot prompt (.github/prompts/*.prompt.md)
  • Copilot agent (.github/agents/*.agent.md)
  • Copilot skill (.github/skills/*/SKILL.md)

Other:

  • Script/automation (.ps1, .sh, .py)
  • Other (please describe):

Sample Prompts (for AI Artifact Contributions)

Testing

Automated testing:

  • Pester test suite: 2002 tests executed — 1935 passed, 0 failed, 2 skipped (7m40s)
  • New test coverage: 19 new tests in Prepare-Extension.Tests.ps1 (+233 lines) and Validate-Collections.Tests.ps1 (+157 lines)
    • Maturity filtering: excludes experimental when stable-only, includes experimental when allowed, excludes deprecated regardless
    • Template markers: preserves intro and replaces marker section, writes back updated artifacts, handles files without markers, preserves footer content
    • Split-CollectionMdByMarkers: no markers, empty string throws, correct intro/footer parsing, partial markers, reversed markers, duplicate markers
  • Lint validation: 100% compliance across markdown, PowerShell, YAML, frontmatter, links, spelling, and skill structure checks
  • Plugin generation: All 14 plugins regenerated successfully — 116 agents, 143 commands, 245 instructions, 30 skills, 0 lint errors

Manual testing: Not performed.

Checklist

Required Checks

  • Documentation is updated (if applicable)
  • Files follow existing naming conventions
  • Changes are backwards compatible (if applicable)
  • Tests added for new functionality (if applicable)

AI Artifact Contributions

Required Automated Checks

The following validation commands must pass before merging:

  • Markdown linting: npm run lint:md
  • Spell checking: npm run spell-check
  • Frontmatter validation: npm run lint:frontmatter
  • Skill structure validation: npm run validate:skills
  • Link validation: npm run lint:md-links
  • PowerShell analysis: npm run lint:ps

@WilliamBerryiii WilliamBerryiii requested a review from a team as a code owner April 7, 2026 23:23
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 7, 2026

Codecov Report

❌ Patch coverage is 61.46789% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.33%. Comparing base (410de7d) to head (5d60dc4).
⚠️ Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
scripts/plugins/Generate-Plugins.ps1 6.66% 42 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1316      +/-   ##
==========================================
- Coverage   87.94%   87.33%   -0.61%     
==========================================
  Files          62       61       -1     
  Lines        9603     9428     -175     
==========================================
- Hits         8445     8234     -211     
- Misses       1158     1194      +36     
Flag Coverage Δ
pester 84.80% <61.46%> (-0.46%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
scripts/collections/Modules/CollectionHelpers.psm1 99.43% <100.00%> (+0.12%) ⬆️
scripts/collections/Validate-Collections.ps1 93.54% <100.00%> (+0.40%) ⬆️
scripts/extension/Prepare-Extension.ps1 94.49% <100.00%> (+0.04%) ⬆️
scripts/plugins/Generate-Plugins.ps1 72.61% <6.66%> (-24.13%) ⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory review — this PR is from a maintainer. Findings are informational only.

Review Summary

The implementation is well-structured and the PR meets quality standards. The description is clear, linked issue is present, type-of-change selections are accurate, and the 19 new Pester tests provide solid coverage of the new behavior. The utf8utf8NoBOM encoding fix in Set-ContentIfChanged is a correct bug fix. The maturity-filtering design is clean and consistent with existing channel logic.

Two advisory observations are raised below.


Issue Alignment ✅

Closes #1256. The PR delivers auto-generation of collection.md artifact tables with channel-based maturity filtering and sentinel-marker support, which matches the stated intent. The write-back behavior, backward-compatible marker validation, and plugin regeneration are all coherent.


PR Template Compliance ✅

Section Status
Description ✅ Complete and detailed
Related Issue Closes #1256
Type of Change ✅ New feature + Script/automation — both correct
Testing ✅ Comprehensive (2002 tests, 0 failures, 19 new tests, lint & plugin generation)
Checklist ✅ All required items checked

Coding Standards

💡 Split-CollectionMdByMarkers — incomplete function decoration (advisory)

Every other function in Prepare-Extension.ps1 uses [CmdletBinding()], [OutputType()], and a full comment block (.DESCRIPTION, .PARAMETER, .OUTPUTS, .EXAMPLE). The PSScriptAnalyzer configuration explicitly sets PSProvideCommentHelp.ExportedOnly = $false, so the convention applies to private functions too. The inline comment above includes a suggested complete signature.

💡 Duplicate marker constants — DRY concern (advisory)

$script:CollectionMdBeginMarker and $script:CollectionMdEndMarker are now defined identically in both Prepare-Extension.ps1 and Validate-Collections.ps1. Since CollectionHelpers.psm1 is imported by both, exporting the constants from there would eliminate the duplication and make any future string change a single-point edit.


Code Quality ✅

No bugs, security issues, or logic errors found. The Split-CollectionMdByMarkers parsing logic handles all the edge cases covered by tests (no markers, partial markers, reversed markers, footer content). The write-back side-effect in New-CollectionReadme is clearly documented in the function's comment block. Test coverage is thorough.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316

Comment thread scripts/collections/Validate-Collections.ps1 Outdated
Comment thread scripts/extension/Prepare-Extension.ps1 Outdated
Copy link
Copy Markdown
Contributor

@katriendg katriendg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, I believe ensuring the plugin:generate takes care of generating the <collectionid>.collection.md file is a good approach, and we could have the Extension prepare regenerate what is needed based on maturity filtering.

Plugins always take everything without maturity filtering, so that in itself so the more I reflect on this, the more I think we need a generation for the plugins (the default), and regenerate one for stable/pre-release when packaging the extension.

Also this will impact the following documentation addition:

🟡 Comment 2 — Documentation gap: contributor guide has no mention of the two-zone collection.md convention

  • File: docs/contributing/ai-artifacts-common.md and .github/copilot-instructions.md
  • Category: Documentation
  • Severity: Should fix before merge

docs/contributing/ai-artifacts-common.md (the primary contributor guide for AI artifacts) extensivelydocuments *.collection.yml structure, maturity values, and plugin generation — but has no mention of:

  1. The <!-- BEGIN AUTO-GENERATED ARTIFACTS --> / <!-- END AUTO-GENERATED ARTIFACTS --> marker format.
  2. That collection.md has two zones (hand-authored intro vs auto-generated table) and contributors must only edit the intro.
  3. What command to run to refresh the artifact table after editing a YAML.
  4. How to seed a new collection with the correct marker structure.

.github/copilot-instructions.md line 104 says "Only edit the intro zone; the artifact table is overwritten by plugin:generate" — this was added during the review session but the underlying plugin:generate wiring gap (Comment 2 above) means this statement is currently inaccurate.

A ### Collection Markdown Auto-Generation subsection should be added to docs/contributing/ai-artifacts-common.md explaining the marker format, the two-zone rule, and the correct command to sync the artifact table. This section should be updated once Comment 2 is resolved so the documented command is accurate.

Comment thread scripts/extension/Prepare-Extension.ps1
Comment thread scripts/extension/Prepare-Extension.ps1 Outdated
Bill Berry added 4 commits April 10, 2026 13:49
…kers

- thread channel and maturity parameters through extension readme generation
- add template markers to collection.md files with backward-compatible processing
- add marker validation rules to collection validation pipeline
- add unit tests for maturity filtering, marker handling, and marker validation

🔧 - Generated by Copilot
- expand New-CollectionReadme help with in-place update and marker behavior (IV-001)
- add ValidateNotNullOrEmpty to AllowedMaturities parameter (IV-002)
- centralize marker strings as script-scope constants in both scripts (IV-003)
- remove unused Existing key from Split-CollectionMdByMarkers return (IV-004)
- add duplicate-BEGIN-marker edge case test (IV-005)
- add empty-string-throws test for Split-CollectionMdByMarkers (IV-006)
- standardize Set-ContentIfChanged encoding to utf8NoBOM (IV-007)

🔧 - Generated by Copilot
…pers

- move marker constants to CollectionHelpers.psm1 and export as module variables
- move Split-CollectionMdByMarkers to CollectionHelpers.psm1 with full decorators
- move Get-ArtifactDescription to CollectionHelpers.psm1 for reuse
- add collection.md writeback block to Generate-Plugins.ps1

🔧 - Generated by Copilot
- regenerate all 14 collection.md files with structured artifact tables
- replace hand-written summaries with auto-generated content between markers
- tables include instructions, agents, prompts, skills per collection

📝 - Generated by Copilot
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory review — this PR is from a maintainer. Findings are informational only.

Review Summary

The refactor successfully closes #1256. The implementation is clean: maturity filtering is channel-driven, the utf8NoBOM encoding fix is correct, and the sentinel-marker write-back design is well-tested. The previous advisory's two blocking DRY concerns (missing [CmdletBinding()] on Split-CollectionMdByMarkers and duplicate marker constants) have both been addressed.

Three advisory observations remain — see inline comments.


Issue Alignment ✅

Closes #1256. Auto-generation of collection.md artifact tables with channel-based maturity filtering and sentinel marker support is fully implemented.


PR Template Compliance ✅

Section Status
Description ✅ Detailed and accurate
Related Issue Closes #1256
Type of Change ✅ New feature + Script/automation
Testing ✅ 19 new Pester tests, 2002 total, 0 failures
Checklist ✅ All required items checked

Coding Standards

Three minor observations are raised as inline comments:

  1. [OutputType([hashtable])] missing on Split-CollectionMdByMarkers — the only function in CollectionHelpers.psm1 without this annotation.
  2. Duplicate allowed-maturity functionsGet-AllowedMaturities (Prepare-Extension.ps1) and Get-AllowedCollectionMaturities (Generate-Plugins.ps1) are functionally identical; consolidation into CollectionHelpers.psm1 is a natural follow-on to the marker-constants refactor.
  3. Artifact-table-building logic duplicated in Generate-Plugins.ps1 relative to New-CollectionReadme — extracting a shared helper would reduce the maintenance surface.

Code Quality ✅

No bugs, logic errors, or security issues found. Maturity filtering is applied correctly in both code paths ($filteredCollection.items is pre-filtered by Select-CollectionItemsByChannel in the plugin path; AllowedMaturities is passed explicitly in the extension-readme path). The Split-CollectionMdByMarkers edge-case handling (reversed markers, partial markers, no markers, empty string) is well-covered by the new tests.

Generated by PR Review for issue #1316 · ● 6.5M

Copy link
Copy Markdown
Contributor

@katriendg katriendg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think is should be good to merge. Just noticed you need to regenerate the plugins.

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

This PR introduces auto-generation of collection.md artifact tables with channel-based maturity filtering and sentinel marker support. The overall direction is sound and the test coverage is thorough. However, there are required template compliance gaps and a functional code quality issue that need to be addressed before merging.


Issue Alignment

Linked issue #1256 is referenced as Closes #1256. The PR description and the scope of changes are self-consistent — the marker-based generation engine, -Channel parameter, maturity filtering, and new Pester tests all coherently address auto-generation of artifact tables. No obvious scope creep or missing requirements were identified based on the PR description.


PR Template Compliance

Two required sections from .github/PULL_REQUEST_TEMPLATE.md are absent from the PR body:

  1. ## Security Considerations section is entirely missing. The template requires three checkboxes under this section:

    • [ ] This PR does not contain any sensitive or NDA information
    • [ ] Any new dependencies have been reviewed for security issues
    • [ ] Security-related scripts follow the principle of least privilege

    These checkboxes must be present and completed.

  2. Two Required Automated Checks are missing from the checklist. The template defines these items which are absent from the PR body:

    • [ ] Plugin freshness: npm run plugin:generate
    • [ ] Docusaurus tests: npm run docs:test

    The PR description does mention under Testing that plugins were regenerated successfully, but the formal checklist items still need to be present and checked.


Coding Standards

scripts/collections/Modules/CollectionHelpers.psm1Split-CollectionMdByMarkers (line 620) is missing [OutputType([hashtable])]. All public module functions must carry an [OutputType] annotation per the repository's PowerShell standards (PSUseOutputTypeCorrectly). See inline comment.

Minor: Split-CollectionMdByMarkers uses the shorthand [Parameter(Mandatory)] while the surrounding codebase convention uses [Parameter(Mandatory = $true)]. Recommend aligning for consistency.


Code Quality

🔴 Markdown table injection (functional bug)

Both Prepare-Extension.ps1:461 and Generate-Plugins.ps1:361 write $entry.Description directly into a markdown table cell without escaping pipe (|) characters. If any artifact's description frontmatter field contains a |, the generated table will be malformed — columns shift and rendered output breaks. The fix is a one-line sanitize before the AppendLine call. See inline comments on both lines.

⚠️ Path-normalization regex in Generate-Plugins.ps1:331

The regex '^\.\//' (matches .//) is inconsistent with '^\./' used in Prepare-Extension.ps1:429 and never matches any current item path. This is a latent defect — see inline comment.

💡 Duplicated artifact table generation logic

The artifact table building block (group items by kind → sort → AppendLine per entry → write back via Set-ContentIfChanged) is implemented identically in both New-CollectionReadme (Prepare-Extension.ps1) and the #region Update collection.md artifact tables block (Generate-Plugins.ps1). Extracting a shared helper such as New-ArtifactTableMarkdown into CollectionHelpers.psm1 would eliminate this duplication and make the pipe-escaping fix a single change. Not a blocker, but a follow-up worth tracking.


Action Items

  1. ❌ Add the ## Security Considerations section with its three checkboxes completed.
  2. ❌ Add and check Plugin freshness: npm run plugin:generate and Docusaurus tests: npm run docs:test in the Required Automated Checks section.
  3. ❌ Add [OutputType([hashtable])] to Split-CollectionMdByMarkers.
  4. ❌ Escape | characters in $entry.Description before writing to markdown table cells — applies to both Prepare-Extension.ps1:461 and Generate-Plugins.ps1:361.
  5. ⚠️ Fix path regex in Generate-Plugins.ps1:331 from '^\.\//' to '^\./'.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316 · ● 3.3M

- refresh plugins/hve-core-all/README.md and plugins/security/README.md after collection.md autogen changes

- pick up new security-review-sbd prompt and owasp-cicd, secure-by-design skills

- update OWASP wording to security skill descriptions

🔒 - Generated by Copilot
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Quality Review

This PR adds auto-generation of collection.md artifact tables from YAML manifests with channel-based maturity filtering and sentinel marker support. The implementation is well-structured, the testing is thorough (19 new tests covering maturity filtering, marker handling, and edge cases), and the encoding fix (utf8NoBOM) is a genuine correctness improvement. Two required fixes are needed before merge.


Issue Alignment

Issue #1256 could not be read directly (integrity policy), but the PR description matches a feature addition for auto-generated collection documentation with maturity filtering. Alignment appears correct based on the description.


PR Template Compliance

✅ Description is complete and detailed.
Closes #1256 linked correctly.
✅ "New feature" and "Script/automation" checkboxes match the actual changes.
✅ Testing section documents automated results thoroughly.
✅ All checklist items checked appropriately.


Coding Standards Findings

1. [ValidateNotNullOrEmpty()] missing on Split-CollectionMdByMarkers.$Content — blocking

See inline comment on CollectionHelpers.psm1 line 623. The test 'Throws for empty string input' expects an exception, but [Parameter(Mandatory)] on [string] does not reject ''; only $null is rejected. Without [ValidateNotNullOrEmpty()] the test fails and the function silently returns a garbage result for empty input.

2. [OutputType([hashtable])] missing on Split-CollectionMdByMarkers — blocking

See inline comment on CollectionHelpers.psm1 line 620. The sibling Get-ArtifactDescription correctly declares [OutputType([string])]; this function must do the same per the repo's enforced PSUseOutputTypeCorrectly PSScriptAnalyzer rule. The .OUTPUTS doc-comment is not a substitute.


Code Quality Findings

3. Regex copy-paste inconsistency in Generate-Plugins.ps1 line 331 — non-blocking

See inline comment on Generate-Plugins.ps1 line 331. The pattern '^\.//' (two slashes) was copied from Prepare-Extension.ps1 but gained an extra /, making it '^\.//' instead of the correct '^\./. It has no runtime impact today because YAML paths don't use ./ prefixes, but the inconsistency will silently break if that convention ever changes.

4. Significant code duplication between Generate-Plugins.ps1 and New-CollectionReadme — suggestion

Lines 318–372 of Generate-Plugins.ps1 duplicate ~50 lines of the artifact-section-building loop that already exists in Prepare-Extension.ps1 (New-CollectionReadme, lines 410–464). The only meaningful difference is that Generate-Plugins.ps1 omits maturity filtering. Extracting a shared Build-ArtifactSections helper into CollectionHelpers.psm1 would eliminate drift risk and make future changes to table format or column layout a single-site edit.


Required Actions

  1. Add [ValidateNotNullOrEmpty()] to the $Content parameter of Split-CollectionMdByMarkers.
  2. Add [OutputType([hashtable])] to Split-CollectionMdByMarkers.
  3. (Optional but recommended) Fix regex in Generate-Plugins.ps1 line 331: '^\.\//' → '^\./'.
  4. (Optional) Extract shared artifact-section builder to reduce duplication.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316 · ● 2.4M

Comment thread scripts/plugins/Generate-Plugins.ps1 Outdated
Comment thread scripts/collections/Modules/CollectionHelpers.psm1
Comment thread scripts/collections/Modules/CollectionHelpers.psm1
- add [OutputType([hashtable])] and [ValidateNotNullOrEmpty()] to Split-CollectionMdByMarkers

- fix regex typo '^\.\//' -> '^\./' in Generate-Plugins.ps1 resolvedPath computation

- regenerate plugin READMEs

🔒 - Generated by Copilot
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory review — this PR is from a maintainer. Findings are informational only.

Review Summary

The implementation is clean and the refactoring trajectory is solid. The three items flagged as blocking in the most recent automated CHANGES_REQUESTED review (#4141595773) — [ValidateNotNullOrEmpty()], [OutputType([hashtable])], and the regex inconsistency — are all confirmed resolved in the current branch (CollectionHelpers.psm1 lines 621–624; Generate-Plugins.ps1 line 331). The previous CHANGES_REQUESTED verdict no longer reflects the current code state.

Two template compliance gaps and two advisory code-quality observations remain.


Issue Alignment ✅

Closes #1256 is referenced. The PR delivers auto-generation of collection.md artifact tables with channel-based maturity filtering, sentinel-marker write-back, UTF-8 NoBOM encoding correction, and 19 new Pester tests. Scope is coherent. No regressions identified.


PR Template Compliance ⚠️

Two required sections from .github/PULL_REQUEST_TEMPLATE.md are absent from the PR body:

1. ## Security Considerations section is missing entirely.
The template requires three checkboxes under this heading:

  • [ ] This PR does not contain any sensitive or NDA information
  • [ ] Any new dependencies have been reviewed for security issues
  • [ ] Security-related scripts follow the principle of least privilege

2. Two Required Automated Checks are not listed in the checklist:

  • [ ] Plugin freshness: npm run plugin:generate
  • [ ] Docusaurus tests: npm run docs:test

Both commands are defined in the template and must appear in the PR body. The Testing section narrative confirms plugins were regenerated successfully, but the formal checklist items still need to be present and ticked.


Previously Blocking Items — Now Resolved ✅

Finding Previous Review Current State
[ValidateNotNullOrEmpty()] missing on Split-CollectionMdByMarkers.$Content #4141595773 blocking ✅ Added at line 624
[OutputType([hashtable])] missing on Split-CollectionMdByMarkers #4141595773 blocking ✅ Added at line 621
Regex '^\.\//' (double-slash) in Generate-Plugins.ps1 #4141595773 non-blocking ✅ Corrected to '^\./' at line 331

Note: the two open review threads for [ValidateNotNullOrEmpty()] and [OutputType] are not yet marked resolved on GitHub, but the implementation confirms they have been addressed.


Coding Standards ✅

All three previously flagged conventions issues are resolved. Split-CollectionMdByMarkers in CollectionHelpers.psm1 now has full comment-based help, [CmdletBinding()], [OutputType([hashtable])], and [ValidateNotNullOrEmpty()]. Get-ArtifactDescription is also well-formed.


Code Quality 💡

Artifact-table-building duplication — See inline comment on Generate-Plugins.ps1 lines 310–376. The added block replicates the item-iteration, StringBuilder table-generation, and marker write-back logic already present in New-CollectionReadme. Extracting a shared Build-CollectionArtifactSection helper into CollectionHelpers.psm1 would give a single place to evolve table format and improve test isolation. Not blocking, but worth tracking as a follow-on.

Test coverage for Generate-Plugins.ps1 new block — Codecov reports 6.66% patch coverage for the 68 lines added to Generate-Plugins.ps1 (42 lines uncovered). The marker-parsing logic is well-covered via unit tests on Split-CollectionMdByMarkers, but the integration path through Invoke-PluginGeneration with markers present is not exercised. A test scenario driving Invoke-PluginGeneration against a fixture with a marker-containing collection.md would close this gap.


Documentation 💡

docs/contributing/ai-artifacts-common.md and .github/copilot-instructions.md still have no mention of the two-zone collection.md convention (hand-authored intro vs. auto-generated artifact table), the marker format, or the command to sync the table after editing a YAML. This was raised by @katriendg in review #4074803431. A short ### Collection Markdown Auto-Generation subsection covering the marker format, the two-zone rule, and npm run plugin:generate as the sync command would prevent contributors from inadvertently editing the auto-generated zone.


Action Items

  1. Add the ## Security Considerations section with all three checkboxes checked.
  2. Add and check npm run plugin:generate and npm run docs:test in the Required Automated Checks list.
  3. (Optional follow-on) Mark the two open review threads for [ValidateNotNullOrEmpty()] and [OutputType] as resolved, since the fixes are in the code.
  4. (Advisory) Track a follow-on issue for extracting Build-CollectionArtifactSection and adding docs/contributing coverage for the two-zone convention.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316 · ● 2.4M

Comment thread scripts/plugins/Generate-Plugins.ps1
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

This PR adds auto-generation of collection.md artifact tables with channel-based maturity filtering. The core feature logic is solid — the marker-delimiter approach, Split-CollectionMdByMarkers, the encoding fix (utf8NoBOM), and the 19 new Pester tests are all well-implemented. However, there are PR template compliance gaps and code duplication issues that need to be addressed before merging.


Issue Alignment

Linked issue #1256 is referenced via Closes #1256. The PR description is detailed and the changes match the stated intent: marker-based write-back, channel filtering, and validation warnings for mismatched markers.


PR Template Compliance

❌ Security Considerations section is missing

The PR template requires a Security Considerations section with three checkboxes:

## Security Considerations
* [ ] This PR does not contain any sensitive or NDA information
* [ ] Any new dependencies have been reviewed for security issues
* [ ] Security-related scripts follow the principle of least privilege

This section is entirely absent from the PR description. Since this PR modifies automation scripts that write files in-place (Set-ContentIfChanged), the third checkbox is directly relevant. Please add the section with checkboxes completed.

❌ Two required automated checks are missing from the checklist

The template's Required Automated Checks section includes entries for npm run plugin:generate and npm run docs:test that are not present in the PR's checklist:

  • [ ] Plugin freshness: npm run plugin:generate
  • [ ] Docusaurus tests: npm run docs:test

The testing section narrative confirms plugin:generate ran successfully (14 plugins regenerated), but these must also appear as checked items in the Required Automated Checks checklist per template requirements.


Coding Standards

⚠️ Code duplication (see inline comment on Generate-Plugins.ps1 lines 310–376)

The artifact table generation block in Generate-Plugins.ps1 is a near-verbatim copy (~65 lines) of the same logic in New-CollectionReadme. This should be extracted into a shared CollectionHelpers.psm1 function.

⚠️ Duplicate channel-mapping functions (see inline comment on Prepare-Extension.ps1 lines 503–527)

Get-AllowedMaturities in Prepare-Extension.ps1 and Get-AllowedCollectionMaturities in Generate-Plugins.ps1 are functionally identical. The shared implementation belongs in CollectionHelpers.psm1.

⚠️ New-CollectionReadme missing [OutputType()] and leaking output (see inline comment on Prepare-Extension.ps1 lines 333–361)

Every other function in Prepare-Extension.ps1 declares [OutputType()]. This function also has an unassigned Set-ContentIfChanged call whose [bool] return value leaks onto the output stream.


Code Quality

💡 Design note — channel defaults write to the same source files

Generate-Plugins.ps1 defaults to Channel = 'PreRelease' when writing collection.md artifact tables, while Prepare-Extension.ps1 defaults to Channel = 'Stable'. Both write to the same *.collection.md files. The committed files in this PR correctly reflect the npm run plugin:generate PreRelease output. However, running Prepare-Extension.ps1 on the Stable channel would silently overwrite those files with a subset of items (stable-only), creating an inconsistent state. Consider documenting which script is the authoritative writer for collection.md, or making the channel default consistent between the two scripts.


Action Items

  1. Add the Security Considerations section to the PR description with appropriate boxes checked.
  2. Add plugin:generate and docs:test to the Required Automated Checks checklist in the PR description.
  3. Extract the duplicated artifact table generation logic into a shared CollectionHelpers.psm1 function.
  4. Consolidate Get-AllowedMaturities / Get-AllowedCollectionMaturities into a single exported function in CollectionHelpers.psm1.
  5. Add [OutputType([void])] to New-CollectionReadme and suppress the Set-ContentIfChanged return value with $null =.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316 · ● 3.8M

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review — feat/collection-md-autogen-maturity-1256

Overview: The PR successfully delivers the sentinel-marker–based auto-generation of collection.md artifact tables with channel-based maturity filtering. The design is coherent, the utf8NoBOM encoding fix is a genuine improvement, and Split-CollectionMdByMarkers + Set-ContentIfChanged are clean, reusable additions to CollectionHelpers.psm1. Two blocking issues need to be addressed before merge.


Issue Alignment

Could not read issue #1256 in full detail, but the PR description ("Closes #1256") and the implementation (marker support, Get-AllowedMaturities filtering, regenerated collection.md/plugin README.md artifacts) are internally consistent. No scope-creep concerns. ✅


PR Template Compliance

All sections filled. Type of change checkboxes are accurate. Testing section documents new test additions. Checklist is complete. ✅


Coding Standards

⚠️ DRY violation: The artifact table generation logic ([System.Text.StringBuilder], foreach ($section in @(...)), | Name | Description | table, sorted entry loop) is copied verbatim from New-CollectionReadme into Generate-Plugins.ps1 lines 346–364. Now that CollectionHelpers.psm1 exists as a shared module, this logic belongs there. See inline comment.

Minor: Get-AllowedCollectionMaturities in Generate-Plugins.ps1 (line 76) and Get-AllowedMaturities in Prepare-Extension.ps1 (line 503) return identical arrays for the same channel input. With CollectionHelpers.psm1 as the natural home, there is an opportunity to consolidate these into one exported function. Non-blocking but worth a follow-up.

Minor: The .PARAMETER Channel help text on Invoke-ExtensionCollectionsGeneration describes the parameter purpose but does not list valid values (Stable, PreRelease), unlike the top-level script parameter block which does enumerate them.


Code Quality

🔴 Blocking — Missing test coverage in Generate-Plugins.ps1 (Codecov: 6.66% patch, 42 lines uncovered): The entire #region Update collection.md artifact tables block is exercised by no existing test. The 19 new tests target New-CollectionReadme and Split-CollectionMdByMarkers exclusively; Generate-Plugins.ps1 has no dedicated test coverage for this new behaviour. The DryRun guard, the happy-path write-back, the marker-absent fallthrough, and the empty-collection case are all untested. See inline comment.

💡 Design observation — Footer positioning in README template: When a collection.md has a footer, it is appended to $bodyForTemplate and rendered into {{BODY}}, which appears before ## Included Artifacts in the README template. Footer content (section headings, notes) would therefore render before the artifact tables. See inline comment. No current collection has a non-empty footer, so no regression today, but the contract is surprising.


Action Items

  1. Add tests for the Generate-Plugins.ps1 collection.md update block — happy path, DryRun guard, no-markers fallthrough, empty items. (Blocking)
  2. Extract the duplicated artifact table builder into a shared CollectionHelpers.psm1 function and call it from both Generate-Plugins.ps1 and New-CollectionReadme. (Blocking — resolves duplication and will naturally bring coverage through the shared module tests)
  3. Document or fix the footer-in-{{BODY}} positioning for future collection authors. (Non-blocking follow-up)

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316 · ● 4.5M

Comment thread scripts/plugins/Generate-Plugins.ps1
Comment thread scripts/plugins/Generate-Plugins.ps1
Comment thread scripts/extension/Prepare-Extension.ps1
bindsi added 2 commits April 23, 2026 14:24
- add synth-data-generate to data science README
- include customer-card-render in experimental README
- document dt-canonical-deck in hve-core-all README

🔧 - Generated by Copilot

Signed-off-by: Marcel Bindseil <[email protected]>
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory review — this PR is from a maintainer. Findings are informational only.

Review Summary

The PR successfully delivers auto-generation of collection.md artifact tables with channel-based maturity filtering and sentinel marker support. The description is clear, the linked issue is present, type-of-change selections accurately reflect the changes, and the 19 new Pester tests provide solid coverage. All prior blocking review threads are marked resolved. The two human reviewers (katriendg, rezatnoMsirhC) have approved.


Issue Alignment

✅ The changes align with the stated goal (auto-generation of collection artifact tables with maturity filtering). The collection.md writeback was correctly added to Generate-Plugins.ps1 so npm run plugin:generate keeps both plugins/*/README.md and collections/*.collection.md in sync.


PR Template Compliance

Two gaps relative to the current .github/PULL_REQUEST_TEMPLATE.md:

⚠️ Security Considerations section is absent. The template requires a dedicated section with three attestation checkboxes:

  • [ ] This PR does not contain any sensitive or NDA information
  • [ ] Any new dependencies have been reviewed for security issues
  • [ ] Security-related scripts follow the principle of least privilege

None of these appear in the PR description. For completeness, these should be filled in before merge.

⚠️ Two Required Automated Checks are not listed in the PR checklist:

  • npm run plugin:generate — plugin generation results are described in the Testing narrative but the checklist item is absent
  • npm run docs:test — no mention of this check in either the checklist or the Testing section

The template now requires both items to be explicitly checked. The author may wish to update the checklist to reflect that plugin:generate was run (evidenced by "All 14 plugins regenerated successfully") and confirm the outcome of docs:test.


Coding Standards

Split-CollectionMdByMarkers in CollectionHelpers.psm1 correctly includes [CmdletBinding()], [OutputType([hashtable])], [ValidateNotNullOrEmpty()], and full comment-based help — a prior blocking violation that has been resolved.

✅ Marker constants ($CollectionMdBeginMarker, $CollectionMdEndMarker) are now centralized in CollectionHelpers.psm1 as exported module variables — another prior violation that has been resolved.


Code Quality

The following observations were raised in prior review threads (all now marked resolved). They are restated here for completeness, as the code changes have not yet landed:

💡 Duplicate artifact table-building logic — The #region Update collection.md artifact tables block in Generate-Plugins.ps1 (lines ~310–376) and the corresponding section in New-CollectionReadme in Prepare-Extension.ps1 implement materially identical StringBuilder + foreach ($section in @(...)) logic. Extracting a shared Build-CollectionArtifactSection helper into CollectionHelpers.psm1 would reduce future table-format drift risk. Non-blocking; the prior resolved thread noted this as a refactoring opportunity.

💡 Footer placement in New-CollectionReadme — When $parsed.HasMarkers is true, $parsed.Footer is concatenated into $bodyForTemplate, which maps to {{BODY}} in the template. Because {{BODY}} precedes {{ARTIFACTS}} in the template order, any future footer content (e.g., ## Prerequisites) will render before the artifact table in the generated README rather than after it. No current collection.md file has a non-empty footer, so there is no visible regression today. A {{FOOTER}} token at the end of the template would make the document order explicit and predictable for future collection authors.

💡 Missing test coverage for Generate-Plugins.ps1 writeback path — The #region Update collection.md artifact tables block has no dedicated Pester tests (no happy-path test, no -DryRun guard test, no empty-items test). The prior review thread flagged this; it remains unaddressed. This does not block the merge but creates a coverage gap for a non-trivial code path.


Action Items

  1. Add the Security Considerations section with appropriate checkbox attestations.
  2. Add plugin:generate and docs:test to the Required Automated Checks checklist (or update the PR description to confirm both passed).
  3. (Optional follow-up) Extract shared artifact table builder and add Generate-Plugins.ps1 writeback tests in a subsequent PR.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #1256 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by PR Review for issue #1316 · ● 1.7M

@bindsi bindsi merged commit 2aee4d6 into main Apr 23, 2026
46 of 47 checks passed
WilliamBerryiii pushed a commit that referenced this pull request Apr 24, 2026
## Pre-Release 3.3.101

### ✨ Features

- add removed maturity tier and retire owasp-docker (#1444)
- add evaluation dataset creator (#1279)
- align RAI planner with guide, remove scoring, improve UX (#1287)
- add PSGallery staleness check and BOM cleanup (#1379)
- ISA-95 network planner agent (#1177)
- auto-generate collection.md with maturity filtering (#1316)
- add folder-consistency check and standardize WARN outp… (#1350)
- add synth-data-generate prompt to data-science collection (#1419)
- add canonical deck workflow and customer-card rendering for design
thinking (#1413)
- add Figma MCP integration for DT artifact export (#1222)
- introduce `owasp-docker` (#1245)
- replace hve-core-specific references with portable discovery-based
language (#1335)
- introduce `owasp-cicd` (#1246)
- add secure-by-design knowledge skill (#1223)
- introduce `owasp-infrastructure` (#1244)
- introduce `owasp-mcp` (#1207)
- add OutputPath parameter to Invoke-LinkLanguageCheck.ps1 (#1229)
- add -OutputPath parameter to Validate-SkillStructure.ps1 (#1225)
- add maintainer-only skip-review label guard (#1293)
- add extension collections overview and integrate into getting started
flow (#950)
- add agentic workflows for automated issue triage, implementation, PR
review, dependency review, and doc-staleness detection (#1219)
- consolidate package-lock.json version sync into
Update-VersionFiles.ps1 (#1240)
- add standards code review agent and full review orchestrator (#1174)
- standardize pytest-mock as Python mocking framework (#1170)
- add Jira backlog workflows and Jira/GitLab skills (#978)
- add centralized version bump script and supply-chain attestation
(#1183)

### 🐛 Bug Fixes

- pin PowerShell-Yaml to 0.4.7 across all install sites (#1378)
- close fork-PR/workflow-file-PR secret-strip gap and normalize
upload-artifact version (#1421)
- replace stream-based lookahead with array indexing in
list-changed-files.sh (#1376)
- centralize ISO 8601 timestamp regex in CIHelpers (#1343)
- update stale documentation date in release-process.md (#1363)
- pin basic-ftp to 5.3.0 to resolve GHSA-rp42-5vxx-qpwr (#1374)
- add bot filter to dependency PR review workflow (#1362)
- resolve pip-audit findings in powerpoint, gitlab, and jira skill lock
files (#1360)
- standardize Timestamp JSON key casing across all lint result files
(#1314)
- add synchronize trigger to PR Review workflow (#1323)
- standardize timestamp in Validate-SkillStructure.ps1 to use
Get-StandardTimestamp (#1280)
- add parallel subagent dispatch and structured JSON contracts to
code-review-full (#1304)
- standardize timestamp in SecurityHelpers.psm1 to use
Get-StandardTimestamp (#1284)
- standardize timestamps in Test-DependencyPinning.ps1 and
SecurityClasses.psm1 (#1282)
- derive collection artifact counts from YAML at build time (#1275)
- standardize timestamp in FrontmatterValidation.psm1 to use
Get-StandardTimestamp (#1285)
- standardize timestamp in Markdown-Link-Check.ps1 to use
Get-StandardTimestamp (#1283)
- escape hyphens in Mermaid diagram on Collections page (#1262)
- add summary timestamp to PSScriptAnalyzer output (#1211)
- fix plugin compatibility and robustness for coding-standards code
review agents (#1289)
- standardize timestamp in Test-CopyrightHeaders.ps1 to use
Get-StandardTimestamp (#1278)
- standardize timestamp in Invoke-YamlLint.ps1 to use
Get-StandardTimestamp (#1270)
- standardize timestamp in Invoke-LinkLanguageCheck.ps1 to use
Get-StandardTimestamp (#1264)
- fix dependency-review path filters and sparse-checkout cone mode
(#1259)
- replace invalid bare tool names with official tool identifiers (#1198)
- fix broken links and remove orphaned reference in code review docs
(#1257)
- exclude Python env dirs from skill validation warnings (#1255)
- pin happy-dom and serialize-javascript to resolve Dependabot
vulnerabilities (#1253)
- remove Mermaid diagram and add missing collection cards (#1247)
- disable MCP servers by default to prevent token limit errors (#1144)
- sync package-lock.json after pre-release version bump (#1236)
- separate mermaid node declarations and add dynamic diagram generation
with tests (#1215)
- replace anchor links in meeting-analyst with bold text references
(#1201)
- remove recursive symlinks in jira and gitlab skill directories (#1233)
- validate-installation scripts now check .github/skills directory
(#1010) (#1206)
- resolve npm audit vulnerabilities via dependency overrides (#1200)
- add post-release triggers to scorecard workflow (#1186)
- add missing .md extensions to relative links in agent documentation
(#1180)

### 📚 Documentation

- broaden Security Review description beyond OWASP (#1385)
- document maintainer advisory mode and skip-review label guard (#1386)
- document ExcludePaths/OutputPath for Invoke-LinkLanguageCheck (#1383)
- CLI getting-started: clarify plugin install commands as alternatives
(-all vs base) (#1251)

### ♻️ Refactoring

- align agent and prompt folder names to collection identifier (#1210)

### 🔧 Maintenance

- pin PSScriptAnalyzer to 1.25.0 and sync stale workflow version
comments (#1389)
- bump lxml from 6.0.2 to 6.1.0 in
/.github/skills/experimental/powerpoint (#1424)
- bump @vscode/vsce from 3.7.1 to 3.9.1 in the npm-dependencies group
(#1390)
- bump the github-actions group across 1 directory with 7 updates
(#1391)
- bump follow-redirects from 1.15.11 to 1.16.0 in /docs/docusaurus
(#1356)
- upgrade Node.js from 20 to 24 and bump cspell to v10 (#1353)
- bump basic-ftp from 5.2.0 to 5.2.1 (#1324)
- update github/gh-aw-actions requirement to
536ea1bad8c6715d098a9dc1afea8d403733acfe in the github-actions group
across 1 directory (#1298)
- update security instruction attributions and compliance (#1294)
- bump the npm-dependencies group with 2 updates (#1297)
- pre-release 3.3.41 (#1252)
- streamline RAI Planner phase structure and documentation (#1273)
- bump happy-dom from 20.8.8 to 20.8.9 in /docs/docusaurus (#1237)
- pre-release 3.3.27 (#1191)
- bump pygments from 2.19.2 to 2.20.0 in /.github/skills/gitlab/gitlab
(#1234)
- bump path-to-regexp from 0.1.12 to 0.1.13 in /docs/docusaurus (#1226)
- bump the github-actions group with 4 updates (#1231)
- add missing folders and alphabetize location lists (#1193)
- bump brace-expansion (#1224)
- bump handlebars from 4.7.8 to 4.7.9 in /docs/docusaurus (#1217)
- bump brace-expansion from 5.0.3 to 5.0.5 in /docs/docusaurus (#1213)
- pre-release 3.3.10 (#1187)
- bump markdownlint-cli2 from 0.21.0 to 0.22.0 in the npm-dependencies
group (#1175)
- bump the github-actions group with 3 updates (#1176)
- pre-release 3.3.1 (#1165)

---
*Managed automatically by pre-release workflow.*

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

collection.md: auto-generation, maturity safety, and custom header support

5 participants