feat(agents): align RAI planner with guide, remove scoring, improve UX#1287
feat(agents): align RAI planner with guide, remove scoring, improve UX#1287WilliamBerryiii merged 46 commits intomainfrom
Conversation
- restructure Phase 2 into binary trigger assessment with T1/T2/T3 tiers
- replace likelihood-impact scoring with restricted-use gate framework
- adopt AI STRIDE extensions with eight AI element types in Phase 4
- unify threat IDs to T-RAI-{NNN} format across all phases
- add rai-sensitive-uses-triggers.instructions.md for Phase 2 depth
- update collection manifests, plugins, and documentation accuracy
- add Sign-RaiArtifacts.ps1 signing script with Pester tests
Closes #1281
🚀 - Generated by Copilot
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1287 +/- ##
==========================================
- Coverage 87.65% 87.62% -0.03%
==========================================
Files 63 65 +2
Lines 9866 10119 +253
==========================================
+ Hits 8648 8867 +219
- Misses 1218 1252 +34
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
First impressions on this draft PR:
The three-part disclaimer formulation exceeds the baseline requirement: (1) covers legal, compliance, and ethics; (2) frames all outputs as suggestions; (3) establishes organizational policy precedence.
Replacing numerical scores with qualitative concern levels and maturity indicators strengthens the disclaimer posture. Qualitative assessments naturally invite human judgment.
Look forward to seeing this in action.
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
Files affected: All files containing the current disclaimer blockquote:
.github/agents/rai-planning/rai-planner.agent.md
.github/instructions/rai-planning/rai-identity.instructions.md (Session Start Display, Exit Point Reminder, State Creation)
.github/prompts/rai-planning/rai-capture.prompt.md
.github/prompts/rai-planning/rai-plan-from-prd.prompt.md
.github/prompts/rai-planning/rai-plan-from-security-plan.prompt.md
Current text:
This tool provides structured prompts and frameworks to support responsible AI planning. It is not a substitute for professional legal, compliance, or ethics review. All outputs are suggestions for human evaluation. Organizational RAI policies and applicable regulations take precedence.
Required text:
This agent is an assistive tool only. It does not provide legal, regulatory, or compliance advice and does not replace Responsible AI review boards, ethics committees, legal counsel, compliance teams, or other qualified human reviewers. The output consists of suggested actions and considerations to support a user's own internal review and decision‑making. All RAI assessments, sensitive use screenings, security models, and mitigation recommendations generated by this tool must be independently reviewed and validated by appropriate legal and compliance reviewers before use. Outputs from this tool do not constitute legal approval, compliance certification, or regulatory sign‑off.
Rationale:
The current disclaimer is too general and does not adequately protect against misinterpretation. The replacement version:
- Enumerates specific professional roles the tool does not replace (RAI review boards, ethics committees, legal counsel, compliance teams)
- Names the specific output types requiring validation (RAI assessments, sensitive use screenings, security models, mitigation recommendations)
- Explicitly excludes certification and sign-off — preventing misinterpretation as approval
- States a mandatory validation requirement ("must be independently reviewed and validated") rather than a softer suggestion
- Addresses regulatory advice directly ("does not provide legal, regulatory, or compliance advice")
- This change should be applied everywhere the current disclaimer blockquote appears, including the Session Start Display and Exit Point Reminder sections in rai-identity.instructions.md which govern when the disclaimer is shown to users.
- update disclaimer in rai-planner agent, three RAI prompt files, and rai-identity instructions - regenerate plugin outputs and reformat doc tables ⚖️ - Generated by Copilot
This has been resolved. |
|
The disclaimer text was strengthened across all existing locations in Given that the new verbatim text explicitly states "Outputs from this tool do not constitute legal approval, compliance certification, or regulatory sign‑off", the handoff document is arguably the most critical place for this disclaimer to appear, since it's the artifact most likely to be shared outside the immediate user session with review boards, legal, or leadership. Suggested change: add between the handoff header metadata and the Work Item Summary. |
Agree and I am addressing this now; but I am still trying to figure out how to do this tastefully. My main area of consideration here is that the handoff artifacts, in their "intended use" are to be persisted locally (not saying they WILL be, but as HVE Core is currently designed, the "intended use" is user specific local storage). Local artifacts, just like GHCP generating code or documentation generation are intended for human review. As currently designed the agent receivers of these handoff artifacts (the backlog agents) must be operationalized by the user explicitly, the artifacts passed to them, and they are backed by 3 operational modalities:
All three operational modalities have multi-step human interaction requirements, and there is no masked workflows happening. The human is identifying and selecting a specific agent for execution, and any delegation (only available to subagents), only has the bounded context of the original agent's human request. I think at the end of the day, the thing we really care about here from a system/agent operations perspective is to ensure that a human has reviewed the artifacts that are intended for other humans to consume. If we can agree on that, then I think the most important thing about AI generation attribution is to provide, as part of the disclaim footer, a markdown checkbox that offers, "has" or "has not" been reviewed by a human, with a default selection of "has not". This sends a much stronger signal, of attestation of review ... which is really what we are trying to set the behavioral outcome for. Consuming agents can also gate on this field ... as an added benefit. |
…system - remove human review checkbox from agentic artifacts (Control Surface Catalog, Evidence Register) in impact assessment instructions - remove superseded qualifier prose from RAI Review Summary template in backlog handoff instructions - add Artifact Attribution and Review section to handoff pipeline docs with footer classification table - add footer classification notes to Phase 5 and Phase 6 outputs in phase reference docs - add conversational vs persisted disclaimer note to agent overview docs 📝 - Generated by Copilot
Artifact Attribution and Review — Summary of ChangesThank you for the thorough review, @raymond-nassar. We implemented a two-tier attribution system across the RAI planning artifact templates and added documentation to support the new conventions. Tier 1: AI-Content NoteAll persisted artifacts now include an AI-content note adapted from the Microsoft Learn per-article pattern:
Tier 2: Full DisclaimerThe handoff summary — the primary stakeholder-facing deliverable — includes the complete verbatim disclaimer after the AI-content note and human review checkbox. Human Review CheckboxHuman-facing artifacts include a review checkbox to track validation status:
Artifact Classification
Control Surface Catalog and Evidence Register are classified as agentic artifacts (consumed by later pipeline phases) and receive only the AI-content note. All other artifacts are human-facing and include the review checkbox. Files ChangedInstruction files:
Documentation:
|
- add centralized config files for disclaimers and footers with human review - add JSON schema for config file validation - add Validate-AIArtifacts.ps1 with scope-filtered artifact classification - add Pester tests (20 passing) for all validation functions - add ai-artifact-validation.yml reusable workflow gated in pr-validation - add lint:ai-artifacts and validate:ai-artifacts npm scripts 🔧 - Generated by Copilot
AI Artifact Validation Infrastructure — Update SummaryThis latest commit ( What was added
Expansion plansThe current scope filter targets
The goal is for each focus area to opt in by adding its scope patterns to the config files — keeping validation centralized while allowing incremental adoption across the codebase. 🔧 - Generated by Copilot |
Correction to previous commentThe scope in Updated scope: Updated expansion plan: When other focus areas (e.g., Validation results after fix: 2 files / 7 issues (down from 3 files / 9 issues), all within 🔧 - Generated by Copilot |
…iling newline Security instruction files were not modified in this PR. Remove .github/instructions/security/** scope patterns from footer-with-review.yml to avoid false positive validation warnings. Fix missing trailing newline in package.json. 🔧 - Generated by Copilot
- switch Find-ArtifactReferences from content-based to filename-based matching - update Pester tests for filename-based artifact matching logic - fix BOM encoding and empty catch blocks for PSScriptAnalyzer - update Docusaurus collection card counts for rai-planning - auto-fix markdown table formatting in handoff-pipeline 🔧 - Generated by Copilot
… consistency - prettify tables in 5 rai-planning instruction files using LF-normalized input - resolves CRLF/LF formatting divergence between Windows and Linux CI 🎨 - Generated by Copilot
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Advisory review, this PR is from a maintainer. Findings are informational only.
Review Summary
This is a substantial, well-structured PR that delivers the RAI Planner redesign described in #1281 alongside a new AI artifact validation pipeline. The implementation quality is high: scripts follow PowerShell and Bash conventions, the new CI workflow is properly permission-scoped with pinned SHA dependencies, and the devcontainer cosign installation includes SHA verification. The testing table confirms all automated checks passed.
The findings below are template compliance and minor quality items.
Issue Alignment
✅ The PR addresses all acceptance criteria from #1281 (standards alignment, disclaimer text, UX improvements, output preferences, phase transitions). The implementation goes substantially beyond the 11-file scope described in the issue — 61 files are changed, including new CI infrastructure, scripts, devcontainer tooling, documentation, plugins, and collections.
.github/agents/rai-planning/, .github/instructions/rai-planning/, and .github/prompts/rai-planning/." The PR adds an entirely new AI artifact validation pipeline, artifact signing infrastructure, DevContainer tooling, and regenerates all 14 plugins. This is well-intentioned and well-documented, but represents significant undisclosed scope growth relative to the linked issue. Future large feature additions may benefit from a separate issue tracking the infrastructure work so reviewers and the changelog can reference it independently.
PR Template Compliance
.github/PULL_REQUEST_TEMPLATE.md) includes a required Security Considerations section with three checkboxes. This section is entirely missing from the PR description:
## Security Considerations
* [ ] This PR does not contain any sensitive or NDA information
* [ ] Any new dependencies have been reviewed for security issues
* [ ] Security-related scripts follow the principle of least privilege
This is especially relevant here since the PR adds scripts/security/Sign-RaiArtifacts.ps1, scripts/linting/Validate-AIArtifacts.ps1, and installs cosign in both the devcontainer and the Copilot coding agent environment.
[ ] I acknowledge this PR includes non-stable GHCP artifacts[ ] Non-stable artifacts are intentional for this change
### AI Artifact Contributions are unchecked. These are listed as required checks for AI artifact contributions:
[ ] Used /prompt-analyze to review contribution[ ] Addressed all feedback from prompt-builder review[ ] Verified contribution follows common standards and type-specific requirements
- Missing from PR checklist:
npm run plugin:generate,npm run docs:test - Present in PR checklist but not in template:
npm run lint:ai-artifacts
The testing table confirms plugin:generate was run and passed — the checklist just doesn't reflect it. Consider updating the PR description to align the Required Automated Checks with the current template, and (separately) the template itself may need updating to include npm run lint:ai-artifacts now that it has been added to lint:all.
GHCP Maturity Table Discrepancy
rai-sensitive-uses-triggers.instructions.md is listed in the GHCP Artifact Maturity table but does not appear in the diff. The diff shows rai-risk-classification.instructions.md as added instead. If rai-sensitive-uses-triggers.instructions.md is an existing file not modified in this PR, it should be removed from the maturity table. If it was renamed, the rename may not be reflected correctly in the table.
PR Size
i️ 61 files were changed. The automated review guideline recommends splitting PRs larger than 50 files to maintain thorough reviewability. For a maintainer-driven feature PR like this, the size is understandable given the coordinated nature of the changes, but it's worth noting for future reference.
Code Quality
✅ Sign-RaiArtifacts.ps1 follows all PowerShell conventions: copyright header, #Requires -Version 7.0, comment-based help, [CmdletBinding()], typed parameters, $ErrorActionPreference = 'Stop', invocation guard, and region blocks.
✅ Validate-AIArtifacts.ps1 follows the same conventions consistently.
✅ on-create.sh cosign installation includes SHA verification consistent with the existing tool installation patterns.
✅ copilot-setup-steps.yml cosign step mirrors the devcontainer installation with SHA verification.
✅ ai-artifact-validation.yml is properly structured as a reusable workflow_call workflow with explicit permissions: contents: read at both workflow and job level, pinned SHA for actions/checkout, and persist-credentials: false.
💡 One inline comment was posted on ai-artifact-validation.yml regarding the unpinned PowerShell-Yaml module version in CI.
Action Items
- Add the Security Considerations section back to the PR description and check the applicable boxes.
- Check the GHCP Maturity Acknowledgment checkboxes.
- Check or address the AI Artifact Contributions checklist items.
- Align the Required Automated Checks checklist with the current template.
- Resolve the
rai-sensitive-uses-triggers.instructions.mdentry in the GHCP Maturity table. - Consider pinning the
PowerShell-Yamlmodule version inai-artifact-validation.yml(and optionally incopilot-setup-steps.ymlandon-create.shfor consistency).
- pin all workflows installing PowerShell-Yaml: ai-artifact-validation, copilot-setup-steps, extension-package, frontmatter-validation, msdate-freshness-check, pester-tests, plugin-package, plugin-validation, release-prerelease-pr, release-stable - pin devcontainer setup: .devcontainer/scripts/on-create.sh - pin docs and error messages: extension/PACKAGING.md, scripts/plugins/README.md, scripts/collections/README.md, Invoke-MsDateFreshnessCheck.ps1, FrontmatterValidation.psm1 - addresses PR #1287 review comment r3104011635 🔒 - Generated by Copilot
There was a problem hiding this comment.
Advisory review, this PR is from a maintainer. Findings are informational only.
PR Review — feat/rai-planner-guide-alignment
⚠️ PR Size Exceeds Reviewable Threshold
This PR contains 61 changed files, which exceeds the 50-file automated review threshold. A thorough automated review cannot be completed. The author is encouraged to consider splitting future large PRs into smaller, focused units to improve reviewability and reduce merge risk.
📋 Issue Alignment
Issue: #1281 — Align RAI Planner with Impact Assessment Guide and improve UX
✅ The PR closes #1281 and addresses the stated goals: guide alignment, UX improvements, disclaimer/attribution, and output preferences.
rai-planning directories. This PR spans 61 files including CI workflows, scripts, plugins, collection metadata, devcontainer config, and a new AI artifact validation pipeline. While the PR description explains these additions clearly, the issue did not capture this broader scope. For transparency, consider updating issue #1281 or creating a follow-on issue to track the AI artifact validation work as a separate deliverable.
📝 PR Template Compliance
✅ Description is well-filled with detailed context, phase descriptions, output artifacts, and success indicators.
✅ Related Issue(s): Closes #1281 — valid reference.
✅ Type of Change: Multiple applicable boxes checked (New feature, Documentation update, GitHub Actions workflow, Security configuration, DevContainer configuration, Dependency update, Copilot instructions/prompts/agents, Script/automation).
❌ GHCP Maturity Acknowledgment checkboxes are unchecked. The PR correctly identifies 11 experimental GHCP artifacts and includes the maturity table, but neither acknowledgment checkbox is checked:
[ ] I acknowledge this PR includes non-stable GHCP artifacts[ ] Non-stable artifacts are intentional for this change
These should be checked before merging.
❌ AI Artifact Contributions checklist is unchecked. All three items remain unchecked:
[ ] Used /prompt-analyze to review contribution[ ] Addressed all feedback from prompt-builder review[ ] Verified contribution follows common standards and type-specific requirements
Per the PR template notes, AI Artifact Contributors are expected to complete the prompt-builder review process. If this was intentionally skipped, add a note explaining why.
❌ Required Automated Checks checklist is unchecked. All 7 validation commands (lint:md, spell-check, lint:frontmatter, validate:skills, lint:md-links, lint:ps, lint:ai-artifacts) remain unchecked in the checklist. The Testing table shows several validations passed, but the checklist items should be checked to confirm each specific command ran successfully.
🔍 Code Quality and Security (Partial — size limit reached)
Due to the 61-file size, a full code quality review was not performed. The items flagged above are based on PR metadata and description only.
📋 Action Items
- ✅ Check the two GHCP Maturity Acknowledgment boxes.
- ✅ Complete or explain the AI Artifact Contributions checklist items.
- ✅ Check the Required Automated Checks boxes after confirming each command passes.
- 💡 Consider whether the AI artifact validation pipeline work warrants its own issue/PR for traceability.
rezatnoMsirhC
left a comment
rezatnoMsirhC
left a comment
There was a problem hiding this comment.
Good PR overall. Phase restructuring, scoring removal, guide alignment, and cosign infrastructure are well executed. Two items need resolution before merge: the indicator method inconsistency in rai-identity.instructions.md and the AI artifact validation scope mismatch that makes the CI job a no-op. Also, the GHCP Maturity table in the PR description references rai-sensitive-uses-triggers.instructions.md but that file was never added; the correct name is rai-risk-classification.instructions.md.
- correct rai-identity indicator types (categorical, continuous) - document footer-with-review.yml artifact scope for CI - resolve Sign-RaiArtifacts artifact dir from git root - hoist PowerShell-Yaml import in Validate-AIArtifacts 🔒 - Generated by Copilot
# Pull Request ## Description Pin the `PowerShell-Yaml` PowerShell Gallery module to `0.4.7` at every install site across the repository so that workflows, the devcontainer, and contributor environments install a deterministic version instead of silently floating to whatever the gallery currently serves. A breaking upstream release would otherwise surface as red CI with no obvious cause. The change applies the same pin pattern (`Install-Module -Name PowerShell-Yaml -RequiredVersion 0.4.7 -Force -Scope CurrentUser`) to: * `.devcontainer/scripts/on-create.sh` * All ten workflows under `.github/workflows/` that install the module (including a re-added `.github/workflows/ai-artifact-validation.yml`) * Documentation snippets in [extension/PACKAGING.md](extension/PACKAGING.md), [scripts/collections/README.md](scripts/collections/README.md), and [scripts/plugins/README.md](scripts/plugins/README.md) * Diagnostic strings in [scripts/linting/Invoke-MsDateFreshnessCheck.ps1](scripts/linting/Invoke-MsDateFreshnessCheck.ps1) and [scripts/linting/Modules/FrontmatterValidation.psm1](scripts/linting/Modules/FrontmatterValidation.psm1) Surfaced from PR #1287 review comment [`r3104011635`](#1287 (comment)). ## Related Issue(s) Closes #1377 Related: #1287 (RAI Planner alignment update — origin of the review feedback) ## Type of Change **Code & Documentation:** * [x] Bug fix (non-breaking change fixing an issue) * [ ] New feature (non-breaking change adding functionality) * [ ] Breaking change (fix or feature causing existing functionality to change) * [ ] Documentation update **Infrastructure & Configuration:** * [x] GitHub Actions workflow * [ ] Linting configuration (markdown, PowerShell, etc.) * [ ] Security configuration * [x] DevContainer configuration * [ ] Dependency update **AI Artifacts:** * [ ] Reviewed contribution with `prompt-builder` agent and addressed all feedback * [ ] Copilot instructions (`.github/instructions/*.instructions.md`) * [ ] Copilot prompt (`.github/prompts/*.prompt.md`) * [ ] Copilot agent (`.github/agents/*.agent.md`) * [ ] Copilot skill (`.github/skills/*/SKILL.md`) **Other:** * [x] Script/automation (`.ps1`, `.sh`, `.py`) * [ ] Other (please describe): ## Testing * Verified diff manually: every previously unpinned `Install-Module PowerShell-Yaml` site now passes `-RequiredVersion 0.4.7`, and idempotency guards (`Get-Module -ListAvailable | Where-Object Version -eq '0.4.7'` / `if (-not (Get-Module ...))`) were preserved or added so reruns stay fast. * Required automated checks listed below. ## Checklist ### Required Checks * [x] Documentation is updated (if applicable) * [x] Files follow existing naming conventions * [x] Changes are backwards compatible (if applicable) * [ ] Tests added for new functionality (if applicable) ### Required Automated Checks * [x] Markdown linting: `npm run lint:md` * [x] Spell checking: `npm run spell-check` * [ ] Frontmatter validation: `npm run lint:frontmatter` * [ ] Skill structure validation: `npm run validate:skills` * [ ] Link validation: `npm run lint:md-links` * [x] PowerShell analysis: `npm run lint:ps` * [ ] Plugin freshness: `npm run plugin:generate` * [ ] Docusaurus tests: `npm run docs:test` ## Security Considerations * [x] This PR does not contain any sensitive or NDA information * [x] Any new dependencies have been reviewed for security issues * [x] Security-related scripts follow the principle of least privilege Pinning a specific module version reduces supply-chain risk by removing implicit trust in whatever version the gallery serves at install time. No new dependencies are introduced; the pinned version (`0.4.7`) is already in active use. ## Additional Notes * `.github/workflows/ai-artifact-validation.yml` is added back as part of this change so the pin can land at every install site uniformly. A merge conflict against `main` is anticipated for this file and will be resolved in a follow-up. * Renovate already monitors the `powershell-galleryV2` ecosystem, so it can propose future bumps once a pinned version is in place. Dependabot does not yet have a PowerShell Gallery datasource.
- re-apply planner rename for validate and sign scripts - sync ai-artifact-validation workflow with renamed planner script - merge cspell, collection, and package.json updates - regenerate plugin READMEs 🔀 - Generated by Copilot
- delete collections/rai-planning.collection.{yml,md}
- remove plugins/rai-planning and extension rai-planning artifacts
- regenerate marketplace manifest (13 plugins)
🧹 - Generated by Copilot
Update: main merge + standalone rai-planning collection removedMerge from main (4fbda7c)Merged
Collection change (418d8e2)Removed the standalone
Validation
Follow-upThe broader 🧹 - Generated by Copilot |
There was a problem hiding this comment.
Advisory review, this PR is from a maintainer. Findings are informational only.
Overview
This PR significantly expands the RAI Planner system with a 5→6 phase restructuring, scoring model replacement, artifact signing infrastructure, and a new config-driven AI artifact validation pipeline. The implementation is thorough and well-documented. Three advisory findings are noted below.
Issue Alignment
Issue #1281 described a scope of 11 files across the rai-planning directories. This PR touches 77 files across agents, instructions, prompts, documentation, plugins, collection manifests, CI workflows, scripts, and the devcontainer. The expanded scope includes work that was not mentioned in the issue — specifically:
- Deletion of the standalone
rai-planningcollection (collections/rai-planning.collection.yml/.md) - New CI validation workflow (
ai-artifact-validation.yml) - Artifact signing script and Pester test suite
- cosign installation in the devcontainer
- Changes to
security-planner.agent.mdandsssc-planner.agent.md
The issue acceptance criteria remain unchecked in the issue itself (though the PR description confirms they were satisfied). Consider closing the issue or updating acceptance criteria as the implementation has clearly moved well beyond the original 11-file scope.
The deletion of collections/rai-planning.collection.yml is a potentially breaking change for users who had the standalone rai-planning collection installed separately. The PR marks no checkboxes for "Breaking change," though this is arguable given the experimental maturity of the artifacts.
PR Template Compliance
Three sets of checkboxes in the PR template are unchecked and should be addressed before merge:
- GHCP Maturity Acknowledgment — Both acknowledgment boxes under "GHCP Artifact Maturity" are unchecked (
[ ] I acknowledge this PR includes non-stable GHCP artifacts/[ ] Non-stable artifacts are intentional for this change). - AI Artifact Contributions checklist — All three items (
/prompt-analyze,prompt-builderreview, and standards verification) are unchecked. - Required Automated Checks — All seven validation commands (
lint:md,spell-check,lint:frontmatter,validate:skills,lint:md-links,lint:ps,lint:ai-artifacts) appear unchecked in the template, though the Testing table confirms they were run and passed.
Additionally, "Linting configuration" was not checked under Infrastructure & Configuration, though Validate-PlannerArtifacts.ps1 is wired into lint:all and constitutes a new linter.
Coding Standards
See the two inline comments for:
scripts/security/Sign-PlannerArtifacts.ps1line 31 — stale.EXAMPLEpath referencing the pre-rename filenameSign-RaiArtifacts.ps1.scripts/linting/Validate-PlannerArtifacts.ps1line 207 —Test-DisclaimerInContentis an exact duplicate ofTest-FooterInContent; consider consolidating into a single helper.
Code Quality
footer-with-review.ymlscope mismatch — See inline comment on line 35–50. The CI workflow enforcing this config will always report 0 findings because instruction files never contain the generated footers. The enforcement gap is documented in the YAML but could mislead future maintainers. Consider a more explicit comment in the CI workflow or a fixture-based test.- PR size (77 files) — While the changes are logically coherent, the PR spans many concern areas (RAI planner restructure, artifact signing, validation infra, collection reorganization). Future maintainers may find it easier to bisect issues if these are separated by concern.
Summary
No blocking issues. The stale example path and duplicate function are minor clean-up items; the template checkbox state and scope mismatch are housekeeping notes. The implementation quality is high across the new scripts, workflows, and instruction files.
mspuckit
left a comment
mspuckit
left a comment
There was a problem hiding this comment.
Run this through the PR Review and did my own quick review. Things look great. Thank you for removing sensitive and restricted use cases from the RAI planner and including the disclaimers. No easy task and appreciate the effort.
…n deletion Also updates SHA staleness test expectations from 4 to 2 release-latest fetches to match reduced tool inventory.
There was a problem hiding this comment.
Advisory review — this PR is from a maintainer. Findings are informational only.
Review Overview
This is a substantial, well-described PR that meaningfully expands the RAI Planner from a 5-phase to a 6-phase workflow, replaces numeric scoring with qualitative concern levels, and establishes a config-driven AI artifact validation pipeline. The implementation quality is generally high: SHA-pinned workflows, verified binary downloads, thorough Pester test coverage, and clear config-driven validation architecture. Findings below are informational for the author's awareness.
Issue Alignment
Linked issue: Closes #1281
The core issue acceptance criteria are addressed (phase restructuring, guide alignment, disclaimer/attribution, UX improvements, output customization). However, issue #1281 scoped the work to 11 files across three directories. This PR changes 80 files spanning agents, instructions, prompts, CI workflows, devcontainer config, collection manifests, plugins, documentation, scripts, and test suites.
The scope expansion is well-justified and thoroughly documented in the PR description, but the original issue has not been updated to reflect the broader intent. Consider updating issue #1281 to capture the full scope of what shipped, or noting this explicitly in the PR.
PR Template Compliance
Several required template sections are incomplete:
Security Considerations section missing — The template includes a Security Considerations section with three checkboxes. This entire section is absent from the PR body:
- This PR does not contain any sensitive or NDA information
- Any new dependencies have been reviewed for security issues
- Security-related scripts follow the principle of least privilege
Given that this PR adds Sign-PlannerArtifacts.ps1 (handles cosign keyless signing), installs cosign in devcontainer and CI, and adds config files under .github/config/, this section is directly applicable and should be completed.
GHCP Maturity Acknowledgment checkboxes unchecked — The PR correctly identifies 11 experimental artifacts in the maturity table, but neither acknowledgment checkbox is checked:
- I acknowledge this PR includes non-stable GHCP artifacts
- Non-stable artifacts are intentional for this change
AI Artifact Contributions checklist unchecked — All three items remain unchecked despite the PR containing agents, prompts, and instructions:
- Used
/prompt-analyzeto review contribution - Addressed all feedback from
prompt-builderreview - Verified contribution follows common standards and type-specific requirements
Required Automated Checks unchecked — The testing table shows strong validation results, but the checklist checkboxes themselves are all unchecked. Additionally, two standard template checks (Plugin freshness: npm run plugin:generate and Docusaurus tests: npm run docs:test) have been replaced with AI artifact validation: npm run lint:ai-artifacts. The substitution is reasonable given the PR's focus, but deviates from the standard template.
Coding Standards
Applicable instruction files reviewed:
.github/instructions/workflows.instructions.md(GitHub Actions conventions).github/instructions/coding-standards/powershell.instructions.md(PowerShell conventions).github/instructions/coding-standards/powershell-tests.instructions.md(Pester conventions).github/instructions/coding-standards/bash.instructions.md(Bash conventions)
GitHub Actions workflows: ✅ All actions use full SHA pins with semantic version comments; explicit permissions: blocks are declared at both workflow and job level; persist-credentials: false is set where applicable; reusable workflows follow the established pattern.
PowerShell scripts: Mostly ✅. Validate-PlannerArtifacts.ps1 follows all conventions (copyright header, #Requires, comment-based help, [CmdletBinding()], $ErrorActionPreference = 'Stop', #region/#endregion, invocation guard). Sign-PlannerArtifacts.ps1 follows the same pattern with one doc bug (see inline comment on line 32).
Pester tests: ✅ Files follow #Requires -Modules Pester before copyright header; BeforeAll dot-sources scripts; -Tag 'Unit' present; $TestDrive used for isolation.
Bash (on-create.sh): ✅ set -euo pipefail; copyright header; checksum verification before installation; consistent error message format.
Code Quality
Two observations (see inline comments for line-level detail):
-
scripts/security/Sign-PlannerArtifacts.ps1line 32 — First.EXAMPLEentry references the old script nameSign-RaiArtifacts.ps1. The second and third examples use the correctSign-PlannerArtifacts.ps1. Simple one-line fix. -
.github/workflows/copilot-setup-steps.yml— The cosign step (new in this PR) uses a defensiveif !guard for checksum verification with explicitexit 1. The pre-existing actionlint step uses an unguarded inline form. The cosign pattern is strictly safer; a follow-up to align actionlint would improve consistency.
Summary
| Category | Verdict |
|---|---|
| Issue alignment | ✅ Core criteria met; scope expansion well-documented but issue not updated |
| PR template compliance | |
| Coding standards | ✅ Workflows, PowerShell, Pester, and Bash all follow conventions |
| Code quality |
No blocking issues identified. The two items requiring author action before merge are the missing Security Considerations section and the unchecked GHCP Maturity Acknowledgment checkboxes.
…erArtifacts example
There was a problem hiding this comment.
Advisory review, this PR is from a maintainer. Findings are informational only.
PR Review: feat(agents): align RAI planner with guide, remove scoring, improve UX
PR #1287 · 80 files changed · Author: WilliamBerryiii (MEMBER)
This is a broad, well-motivated PR that ships a cohesive set of improvements to the RAI planning agent, adds AI artifact signing and validation infrastructure, and cleans up associated collection manifests and plugin outputs. The implementation quality is generally high. The notes below are informational and do not block merging.
📋 Issue Alignment
Linked issue: Closes #1281
Issue #1281 described a focused change to "11 files across .github/agents/rai-planning/, .github/instructions/rai-planning/, and .github/prompts/rai-planning/." This PR delivers 80 changed files encompassing new CI workflows, PowerShell signing/validation scripts, Pester tests, package.json scripts, collections/ YAML and Markdown, plugins/ regeneration, and documentation.
The scope expansion is substantive and well-justified by the work being done, but issue #1281's description is now stale — it does not capture the CI infrastructure, signing pipeline, or collection/plugin changes that form a significant portion of the PR. Consider updating the issue body or closing it with a comment that summarizes the actual delivery scope, so the history remains accurate for future contributors.
No scope creep in the negative sense — the additional work is clearly additive and cohesive with the stated goal.
📝 PR Template Compliance
The PR description is detailed and well-written. Two template compliance gaps were identified:
1. GHCP Maturity Acknowledgment checkboxes are unchecked
The GHCP Maturity section was inserted (correctly — the PR includes experimental and preview artifacts) but both acknowledgment boxes remain [ ]:
- [ ] I acknowledge this PR includes non-stable GHCP artifacts
- [ ] Non-stable artifacts are intentional for this change
```
These should be checked before merge to confirm deliberate intent.
**2. AI Artifact Contributions checklist is unchecked**
The PR checks `[x] Copilot instructions`, `[x] Copilot prompt`, and `[x] Copilot agent` under Type of Change, which triggers the requirement (from PR template instructions) to also complete the AI Artifact Contributions checklist. All three items are currently `[ ]`:
```
- [ ] Used /prompt-analyze to review contribution
- [ ] Addressed all feedback from prompt-builder review
- [ ] Verified contribution follows common standards and type-specific requirements
Check these or explicitly mark them N/A with a note if the prompt-builder review step was intentionally skipped.
🔍 PR Description Accuracy
Three file name discrepancies between the PR description and the actual committed files:
| PR Description Says | Actual File |
|---|---|
Validate-AIArtifacts.ps1 (612 lines) |
Validate-PlannerArtifacts.ps1 (636 lines) |
Sign-RaiArtifacts.ps1 |
Sign-PlannerArtifacts.ps1 |
"Created ai-artifact-validation.yml" |
File already exists on main with identical SHA — the PR calls this existing workflow from pr-validation.yml, but did not create it |
These don't affect the code, but keeping the description accurate helps reviewers and future readers correlate what was shipped.
🏗️ Coding Standards
.github/workflows/ compliance — ✅ No violations
- Third-party actions are SHA-pinned with version comments (
actions/checkout@de0fac...,actions/upload-artifact@043fb4...) ✅ - Top-level and job-level
permissions:blocks present ✅ persist-credentials: falsepresent ✅- Runners:
ubuntu-latest✅
scripts/security/Sign-PlannerArtifacts.ps1 — ✅ Minor advisory
- Copyright header, shebang,
#Requires -Version 7.0,[CmdletBinding()],$ErrorActionPreference = 'Stop', main invocation guard — all present ✅ Get-ArtifactHashhas abbreviated comment-based help (see inline comment, line 80)
scripts/linting/Validate-PlannerArtifacts.ps1 — ✅
- 636-line script follows all structural conventions ✅
🔒 Code Quality and Security
ai-artifact-validation.yml — Install-Module without checksum
The workflow installs [email protected] using Install-Module -RequiredVersion 0.4.7 -Force without hash/checksum verification. This contrasts with the cosign installation in copilot-setup-steps.yml which explicitly verifies a SHA256 digest after download.
For a module installed at CI runtime to validate artifact integrity, an unverified module installation is a minor inconsistency in the security posture. This is a low-severity advisory observation — PowerShell Gallery packages are not known malicious, and RequiredVersion pins the exact release — but if the validation workflow's trust model is important, consider pinning the module install to a verified hash (e.g., downloading the .nupkg and verifying before expanding).
copilot-setup-steps.yml — Redundant version print
cosign version is called at the end of the "Install cosign" step and again in a separate "Verify tool availability" step. This is harmless but generates duplicate output in CI logs. Consider removing the redundant call from the install step.
📏 PR Size Advisory
This PR changes 80 files, which exceeds the 50-file threshold noted in the review guidelines. The changes are logically cohesive and the scope expansion from the original issue is justified, so this is purely informational. Future PRs of this scope could consider sequencing infrastructure additions (CI workflows, signing) ahead of content changes in a separate preparatory PR for easier review.
✅ Summary
Overall this is a solid, well-structured PR. The RAI planner alignment looks coherent, the new signing/validation infrastructure follows established repository patterns, and the CI additions are correctly implemented. The items above are minor and informational:
- Check GHCP Maturity Acknowledgment boxes in the PR description
- Check or annotate AI Artifact Contributions checklist items
- Correct script names in the PR description (
Validate-PlannerArtifacts.ps1,Sign-PlannerArtifacts.ps1) and clarify theai-artifact-validation.ymlclaim - Update issue #1281 to reflect the actual delivery scope
- See inline comments for two minor code-level observations
| "extension:package:prerelease": "pwsh ./scripts/extension/Package-Extension.ps1 -PreRelease", | ||
| "validate:copyright": "pwsh -File scripts/linting/Test-CopyrightHeaders.ps1", | ||
| "validate:skills": "pwsh -NoProfile -Command \"& './scripts/linting/Validate-SkillStructure.ps1' -WarningsAsErrors\"", | ||
| "validate:ai-artifacts": "pwsh -NoProfile -Command \"& './scripts/linting/Validate-PlannerArtifacts.ps1'\"", |
There was a problem hiding this comment.
💡 Advisory: Disambiguation between validate:ai-artifacts and lint:ai-artifacts
Both scripts invoke Validate-PlannerArtifacts.ps1, but with different semantics:
lint:ai-artifacts(inlint:all) — runs with-FailOnMissing, enforcing CI compliancevalidate:ai-artifacts(standalone) — runs without-FailOnMissing, informational pre-check
The distinction is sound, but may not be immediately clear to contributors who see both validate:skills and validate:ai-artifacts and expect them to behave similarly (both would appear to be soft-check aliases of their lint: counterparts, but validate:skills is in lint:all via a different route).
Consider adding a brief comment in CONTRIBUTING.md or the npm scripts section of the project instructions to document which variant to use locally vs. in CI. The validate:* prefix elsewhere in the repo (validate:copyright, validate:skills) is not consistently "informational-only", so the naming pattern could be confusing.
| param( | ||
| [Parameter(Mandatory)] | ||
| [string]$FilePath | ||
| ) |
There was a problem hiding this comment.
💡 Advisory: Incomplete comment-based help in Get-ArtifactHash
Per the PowerShell Script Instructions (applyTo: **/*.ps1):
Use block comment style with
.SYNOPSIS,.DESCRIPTION,.PARAMETER,.EXAMPLE, and.NOTESsections
The Get-ArtifactHash function's help block is missing the .PARAMETER FilePath entry and an .EXAMPLE section. PSScriptAnalyzer's PSProvideCommentHelp rule enforces comment-based help for functions with parameters.
Suggested addition:
<#
.SYNOPSIS
Computes the SHA-256 hash of a file and returns a lowercase hex string.
.PARAMETER FilePath
Absolute path to the file to hash.
.OUTPUTS
[string] Lowercase hex SHA-256 digest.
.EXAMPLE
Get-ArtifactHash -FilePath '/tmp/report.json'
#>## Pre-Release 3.3.101 ### ✨ Features - add removed maturity tier and retire owasp-docker (#1444) - add evaluation dataset creator (#1279) - align RAI planner with guide, remove scoring, improve UX (#1287) - add PSGallery staleness check and BOM cleanup (#1379) - ISA-95 network planner agent (#1177) - auto-generate collection.md with maturity filtering (#1316) - add folder-consistency check and standardize WARN outp… (#1350) - add synth-data-generate prompt to data-science collection (#1419) - add canonical deck workflow and customer-card rendering for design thinking (#1413) - add Figma MCP integration for DT artifact export (#1222) - introduce `owasp-docker` (#1245) - replace hve-core-specific references with portable discovery-based language (#1335) - introduce `owasp-cicd` (#1246) - add secure-by-design knowledge skill (#1223) - introduce `owasp-infrastructure` (#1244) - introduce `owasp-mcp` (#1207) - add OutputPath parameter to Invoke-LinkLanguageCheck.ps1 (#1229) - add -OutputPath parameter to Validate-SkillStructure.ps1 (#1225) - add maintainer-only skip-review label guard (#1293) - add extension collections overview and integrate into getting started flow (#950) - add agentic workflows for automated issue triage, implementation, PR review, dependency review, and doc-staleness detection (#1219) - consolidate package-lock.json version sync into Update-VersionFiles.ps1 (#1240) - add standards code review agent and full review orchestrator (#1174) - standardize pytest-mock as Python mocking framework (#1170) - add Jira backlog workflows and Jira/GitLab skills (#978) - add centralized version bump script and supply-chain attestation (#1183) ### 🐛 Bug Fixes - pin PowerShell-Yaml to 0.4.7 across all install sites (#1378) - close fork-PR/workflow-file-PR secret-strip gap and normalize upload-artifact version (#1421) - replace stream-based lookahead with array indexing in list-changed-files.sh (#1376) - centralize ISO 8601 timestamp regex in CIHelpers (#1343) - update stale documentation date in release-process.md (#1363) - pin basic-ftp to 5.3.0 to resolve GHSA-rp42-5vxx-qpwr (#1374) - add bot filter to dependency PR review workflow (#1362) - resolve pip-audit findings in powerpoint, gitlab, and jira skill lock files (#1360) - standardize Timestamp JSON key casing across all lint result files (#1314) - add synchronize trigger to PR Review workflow (#1323) - standardize timestamp in Validate-SkillStructure.ps1 to use Get-StandardTimestamp (#1280) - add parallel subagent dispatch and structured JSON contracts to code-review-full (#1304) - standardize timestamp in SecurityHelpers.psm1 to use Get-StandardTimestamp (#1284) - standardize timestamps in Test-DependencyPinning.ps1 and SecurityClasses.psm1 (#1282) - derive collection artifact counts from YAML at build time (#1275) - standardize timestamp in FrontmatterValidation.psm1 to use Get-StandardTimestamp (#1285) - standardize timestamp in Markdown-Link-Check.ps1 to use Get-StandardTimestamp (#1283) - escape hyphens in Mermaid diagram on Collections page (#1262) - add summary timestamp to PSScriptAnalyzer output (#1211) - fix plugin compatibility and robustness for coding-standards code review agents (#1289) - standardize timestamp in Test-CopyrightHeaders.ps1 to use Get-StandardTimestamp (#1278) - standardize timestamp in Invoke-YamlLint.ps1 to use Get-StandardTimestamp (#1270) - standardize timestamp in Invoke-LinkLanguageCheck.ps1 to use Get-StandardTimestamp (#1264) - fix dependency-review path filters and sparse-checkout cone mode (#1259) - replace invalid bare tool names with official tool identifiers (#1198) - fix broken links and remove orphaned reference in code review docs (#1257) - exclude Python env dirs from skill validation warnings (#1255) - pin happy-dom and serialize-javascript to resolve Dependabot vulnerabilities (#1253) - remove Mermaid diagram and add missing collection cards (#1247) - disable MCP servers by default to prevent token limit errors (#1144) - sync package-lock.json after pre-release version bump (#1236) - separate mermaid node declarations and add dynamic diagram generation with tests (#1215) - replace anchor links in meeting-analyst with bold text references (#1201) - remove recursive symlinks in jira and gitlab skill directories (#1233) - validate-installation scripts now check .github/skills directory (#1010) (#1206) - resolve npm audit vulnerabilities via dependency overrides (#1200) - add post-release triggers to scorecard workflow (#1186) - add missing .md extensions to relative links in agent documentation (#1180) ### 📚 Documentation - broaden Security Review description beyond OWASP (#1385) - document maintainer advisory mode and skip-review label guard (#1386) - document ExcludePaths/OutputPath for Invoke-LinkLanguageCheck (#1383) - CLI getting-started: clarify plugin install commands as alternatives (-all vs base) (#1251) ### ♻️ Refactoring - align agent and prompt folder names to collection identifier (#1210) ### 🔧 Maintenance - pin PSScriptAnalyzer to 1.25.0 and sync stale workflow version comments (#1389) - bump lxml from 6.0.2 to 6.1.0 in /.github/skills/experimental/powerpoint (#1424) - bump @vscode/vsce from 3.7.1 to 3.9.1 in the npm-dependencies group (#1390) - bump the github-actions group across 1 directory with 7 updates (#1391) - bump follow-redirects from 1.15.11 to 1.16.0 in /docs/docusaurus (#1356) - upgrade Node.js from 20 to 24 and bump cspell to v10 (#1353) - bump basic-ftp from 5.2.0 to 5.2.1 (#1324) - update github/gh-aw-actions requirement to 536ea1bad8c6715d098a9dc1afea8d403733acfe in the github-actions group across 1 directory (#1298) - update security instruction attributions and compliance (#1294) - bump the npm-dependencies group with 2 updates (#1297) - pre-release 3.3.41 (#1252) - streamline RAI Planner phase structure and documentation (#1273) - bump happy-dom from 20.8.8 to 20.8.9 in /docs/docusaurus (#1237) - pre-release 3.3.27 (#1191) - bump pygments from 2.19.2 to 2.20.0 in /.github/skills/gitlab/gitlab (#1234) - bump path-to-regexp from 0.1.12 to 0.1.13 in /docs/docusaurus (#1226) - bump the github-actions group with 4 updates (#1231) - add missing folders and alphabetize location lists (#1193) - bump brace-expansion (#1224) - bump handlebars from 4.7.8 to 4.7.9 in /docs/docusaurus (#1217) - bump brace-expansion from 5.0.3 to 5.0.5 in /docs/docusaurus (#1213) - pre-release 3.3.10 (#1187) - bump markdownlint-cli2 from 0.21.0 to 0.22.0 in the npm-dependencies group (#1175) - bump the github-actions group with 3 updates (#1176) - pre-release 3.3.1 (#1165) --- *Managed automatically by pre-release workflow.* Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
7 participants
Description
Implements Issue #1281: RAI Planner Updates — Guide Alignment, Scoring Removal, and UX Improvements.
The RAI Planner agent expands from a 5-phase to a 6-phase workflow, replacing numeric scoring with qualitative assessment, aligning terminology and structure with the Microsoft Responsible AI Impact Assessment Guide, and introducing artifact signing infrastructure. Additionally, this PR establishes a config-driven AI artifact validation pipeline for enforcing footer and disclaimer standards across instruction files. 46 files changed across agent definitions, instructions, prompts, documentation, plugins, collection metadata, config, CI workflows, and scripts.
Key Changes
Basic/Standard/Comprehensive). Subsequent phases renumbered.Low/Moderate/High), a review quality checklist, maturity indicators, and audience adaptation profiles. Renamedrai-scorecard.md→rai-review-summary.md.T-RAI-{NNN}.Sign-RaiArtifacts.ps1for SHA-256 manifest generation with optional Sigstore cosign keyless signing, backed by 262-line Pester test suite. Installed cosign v3.0.5 in devcontainer..github/config/disclaimers.yml,.github/config/footer-with-review.yml) defining footer text, human review checkboxes, and tiered artifact classification with scope-aware glob patterns. CreatedValidate-AIArtifacts.ps1(612 lines) for CI enforcement, a JSON Schema for config validation, a reusable GitHub Actions workflow (ai-artifact-validation.yml), and a 20-test Pester suite. Wired intopr-validation.ymland thelint:allnpm chain.Related Issue(s)
Closes #1281
Type of Change
Select all that apply:
Code & Documentation:
Infrastructure & Configuration:
AI Artifacts:
prompt-builderagent and addressed all feedback.github/instructions/*.instructions.md).github/prompts/*.prompt.md).github/agents/*.agent.md).github/skills/*/SKILL.md)> Note for AI Artifact Contributors:
>
> * Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review
.github/agents/before creating new ones.> * Skills: Must include both bash and PowerShell scripts. See Skills.
> * Model Versions: Only contributions targeting the latest Anthropic and OpenAI models will be accepted. Older model versions (e.g., GPT-3.5, Claude 3) will be rejected.
> * See Agents Not Accepted and Model Version Requirements.
Other:
.ps1,.sh,.py)Sample Prompts (for AI Artifact Contributions)
User Request:
Invoke
RAI Plannerin the VS Code chat pane and use one of three entry prompts:/rai-capture— Start a new conversational RAI assessment from scratch/rai-plan-from-prd— Generate an RAI plan from an existing PRD/rai-plan-from-security-plan— Generate an RAI plan from an existing security planExecution Flow:
T-RAI-{NNN}threats.rai-review-summary.md, dual-format backlog (ADO + GitHub), and optional signed artifact manifest.Output Artifacts:
.copilot-tracking/rai-plans/{session}/state.json— Session state with phase progression.copilot-tracking/rai-plans/{session}/rai-review-summary.md— Qualitative review summary (replaces scored scorecard).copilot-tracking/rai-plans/{session}/rai-backlog-*.md— Dual-format work item backlog.copilot-tracking/rai-plans/{session}/rai-manifest.json— SHA-256 artifact manifest (optional signing)Success Indicators:
rai-review-summary.mdinstead ofrai-scorecard.mdT-RAI-{NNN}format consistentlyTesting
npm run plugin:generate) — 14 pluginsnpm run lint:ai-artifacts> Note: Manual testing was performed along side automated validation and sandbox evaluation as the primary verification methods.
GHCP Artifact Maturity
> [!WARNING]
> This PR includes experimental GHCP artifacts that may have breaking changes.
.github/agents/rai-planning/rai-planner.agent.md.github/prompts/rai-planning/rai-capture.prompt.md.github/prompts/rai-planning/rai-plan-from-prd.prompt.md.github/prompts/rai-planning/rai-plan-from-security-plan.prompt.md.github/instructions/rai-planning/rai-identity.instructions.md.github/instructions/rai-planning/rai-standards.instructions.md.github/instructions/rai-planning/rai-security-model.instructions.md.github/instructions/rai-planning/rai-impact-assessment.instructions.md.github/instructions/rai-planning/rai-backlog-handoff.instructions.md.github/instructions/rai-planning/rai-capture-coaching.instructions.md.github/instructions/rai-planning/rai-sensitive-uses-triggers.instructions.mdGHCP Maturity Acknowledgment
Checklist
Required Checks
AI Artifact Contributions
/prompt-analyzeto review contributionprompt-builderreviewRequired Automated Checks
The following validation commands must pass before merging:
npm run lint:mdnpm run spell-checknpm run lint:frontmatternpm run validate:skillsnpm run lint:md-linksnpm run lint:psnpm run lint:ai-artifacts> Repository template used:
.github/PULL_REQUEST_TEMPLATE.md