Skip to content

ci: default maturity evidence to all profile#96595

Merged
RomneyDa merged 1 commit into
mainfrom
qa-profile-evidence-all-default
Jun 25, 2026
Merged

ci: default maturity evidence to all profile#96595
RomneyDa merged 1 commit into
mainfrom
qa-profile-evidence-all-default

Conversation

@RomneyDa

Copy link
Copy Markdown
Member

What Problem This Solves

The QA Profile Evidence workflow defaulted manual dispatches to the narrower release profile, while the maturity scorecard refresh path is intended to use full taxonomy evidence.

Why This Change Was Made

  • Default manual QA Profile Evidence dispatches to all.
  • Have the maturity scorecard workflow generate and validate all profile evidence.
  • Update workflow guard tests so generated evidence and manifest validation stay aligned.

User Impact

Maintainers refreshing the maturity scorecard get full taxonomy QA evidence by default instead of accidentally rendering from the narrower release profile.

Evidence

  • node scripts/run-vitest.mjs test/scripts/ci-workflow-guards.test.ts
  • pnpm exec oxfmt --check --threads=1 test/scripts/ci-workflow-guards.test.ts
  • actionlint .github/workflows/qa-profile-evidence.yml .github/workflows/maturity-scorecard.yml
  • git diff --check

@clawsweeper

clawsweeper Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 24, 2026, 7:41 PM ET / 23:41 UTC.

Summary
The PR changes QA Profile Evidence manual dispatch, maturity-scorecard generated evidence, manifest validation, PR-body copy, and guard-test expectations from the release profile to all.

PR surface: Tests +3, Config 0. Total +3 across 3 files.

Reproducibility: not applicable. this is a CI workflow/admin contract change, not a runtime bug. Source inspection confirms current main still uses release, and the PR diff switches the workflow path to all.

Review metrics: 1 noteworthy metric.

  • QA profile workflow contract: 1 default changed, 1 caller changed, 2 validators changed. These release-to-all contract changes affect maintainer evidence runs and artifact compatibility before merge.

Stored data model
Persistent data-model change detected: serialized state: .github/workflows/maturity-scorecard.yml. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Confirm the broader all profile default and release-artifact compatibility change are acceptable before merge.

Risk before merge

  • [P1] Changing qa_profile from release to all makes the maintainer-facing QA evidence path run every active taxonomy category, which can broaden run time and failure surface beyond what normal unit checks prove.
  • [P1] The maturity-scorecard validator will reject provided qa_evidence_run_id artifacts whose evidence or manifest profile is still release, so maintainers should accept that artifact compatibility change before merge.

Maintainer options:

  1. Land the all-profile contract (recommended)
    Land as-is if maintainers want manual QA Profile Evidence defaults and maturity-scorecard generated/provided evidence validation to require full-taxonomy runs.
  2. Preserve release artifact compatibility
    Adjust the workflow before merge if existing release-profile evidence run IDs should remain valid inputs for maturity-scorecard rendering.
  3. Pause for live workflow cost review
    Pause the PR if the broader live all profile runtime, secret-bearing workflow usage, or failure surface is not yet accepted as the default maintainer path.

Next step before merge

  • No automated repair is needed; maintainers should decide whether the full-taxonomy workflow default and release-profile artifact rejection are intentional before merge.

Security
Cleared: No concrete security or supply-chain issue was found; the workflow diff changes internal QA profile values and validation text without broadening permissions, secrets, dependencies, or third-party action refs.

Review details

Best possible solution:

Land the branch only after maintainers explicitly accept the full-taxonomy evidence contract; keep or add release-profile artifact compatibility if that operator path is still needed.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a CI workflow/admin contract change, not a runtime bug. Source inspection confirms current main still uses release, and the PR diff switches the workflow path to all.

Is this the best way to solve the issue?

Yes, with maintainer acceptance. The diff keeps caller, validators, generated copy, and guard tests aligned; the safer alternative is to preserve release-profile artifact compatibility if maintainers still need that path.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3ab8d6aa609a.

Label changes

Label justifications:

  • P3: This is a low-risk maintainer CI/admin workflow change rather than an urgent user-facing runtime regression.
  • merge-risk: 🚨 automation: The PR changes GitHub workflow defaults and maturity-scorecard evidence validation in a way green unit checks do not fully prove for live full-taxonomy QA runs.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Not applicable: this is a maintainer-labeled PR, so the external contributor real-behavior proof gate does not apply; the PR body and CI provide supplemental workflow validation.
Evidence reviewed

PR surface:

Tests +3, Config 0. Total +3 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 1 4 1 +3
Docs 0 0 0 0
Config 2 8 8 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 12 9 +3

What I checked:

  • Protected maintainer PR: Live PR metadata shows the maintainer label and MEMBER author association, so cleanup close is not appropriate even though the branch is focused. (834fe34d7ad8)
  • Current manual default is release: Current main still defaults the manual QA Profile Evidence qa_profile input to release; the PR changes this default to all. (.github/workflows/qa-profile-evidence.yml:21, 3ab8d6aa609a)
  • Current maturity scorecard uses release evidence: Current main calls QA Profile Evidence with qa_profile: release and rejects downloaded evidence unless the evidence and manifest resolve to release; the PR changes those checks to all. (.github/workflows/maturity-scorecard.yml:137, 3ab8d6aa609a)
  • All profile is broader than release: The taxonomy defines all as every active category and describes it as intended for explicit QA evidence dispatches rather than the default release gate, making the default change maintainer-facing automation policy. (taxonomy.yaml:218, 3ab8d6aa609a)
  • Renderer already prefers all evidence: The maturity docs renderer checks all evidence before falling back to release, so the PR aligns the workflow-generated evidence with the renderer's preferred coverage source. (scripts/qa/render-maturity-docs.ts:719, 3ab8d6aa609a)
  • Guard coverage follows the workflow contract: The adjacent workflow guard test currently asserts the maturity-scorecard handoff and QA evidence paths; the PR updates that guard to pin the new all contract. (test/scripts/ci-workflow-guards.test.ts:629, 3ab8d6aa609a)

Likely related people:

  • RomneyDa: Recent merged work by this handle built the generic QA Profile Evidence workflow, maturity-scorecard evidence inputs, the all taxonomy profile, and coverage-score rendering that this PR continues. (role: recent area contributor; confidence: high; commits: f6674da87290, 175f493d48d1, dff07f158f97; files: .github/workflows/qa-profile-evidence.yml, .github/workflows/maturity-scorecard.yml, taxonomy.yaml)
  • Shakker: Available local blame for the current workflow default and maturity-scorecard release validation points to the grafted base commit; the shallow/grafted history makes this a routing hint rather than complete ownership proof. (role: introduced behavior in available local history; confidence: low; commits: 8ecbf83c678c; files: .github/workflows/qa-profile-evidence.yml, .github/workflows/maturity-scorecard.yml, test/scripts/ci-workflow-guards.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jun 24, 2026
@RomneyDa
RomneyDa merged commit 8a5cb85 into main Jun 25, 2026
124 of 132 checks passed
@RomneyDa
RomneyDa deleted the qa-profile-evidence-all-default branch June 25, 2026 00:32
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 25, 2026
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant