Skip to content

Convert QA scenarios to YAML files#92915

Merged
RomneyDa merged 3 commits into
mainfrom
scenario-yaml
Jun 15, 2026
Merged

Convert QA scenarios to YAML files#92915
RomneyDa merged 3 commits into
mainfrom
scenario-yaml

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jun 14, 2026

Copy link
Copy Markdown
Member

Summary

  • Convert the QA scenario corpus from Markdown files with fenced qa-scenario / qa-flow YAML blocks to direct .yaml files with top-level title, scenario, and flow keys.
  • Replace the qa-lab fenced-block parsing machinery with direct YAML file parsing, and update the exported qa-lab helpers plus generated workspace artifact to QA_SCENARIOS.yaml.
  • Update scenario references, QA docs, tests, and the QA testing skill to point at qa/scenarios/**/*.yaml and qa/scenarios/index.yaml.

Verification

  • git diff --check
  • node scripts/run-vitest.mjs extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/scenario-packs.test.ts extensions/qa-lab/src/scenario-flow-runner.test.ts extensions/qa-lab/src/suite-runtime-flow.test.ts extensions/qa-lab/src/qa-agent-workspace.test.ts extensions/qa-lab/src/docker-harness.test.ts extensions/qa-lab/src/tool-coverage-report.test.ts
  • pnpm build
  • .agents/skills/autoreview/scripts/autoreview --mode local initially found stale qa-lab API barrel exports; those were fixed. A fresh rerun is currently in progress.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation cli CLI command changes extensions: qa-lab size: XL maintainer Maintainer-authored PR labels Jun 14, 2026
@clawsweeper

clawsweeper Bot commented Jun 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 14, 2026, 4:03 AM ET / 08:03 UTC.

Summary
The PR converts the qa-lab scenario corpus from Markdown files with fenced YAML blocks to direct YAML files, updates the scenario catalog parser and qa-lab API exports, and refreshes related docs/tests/QA skill references.

Reproducibility: not applicable. This PR is a maintainer refactor of QA scenario source format, not a bug report with a failing user path. The relevant check is whether the catalog loads the converted YAML scenarios and the adjacent tests cover that path.

Review metrics: 1 noteworthy metric.

  • Private QA helper surface: 2 exported helpers replaced. The qa-lab API barrel swaps Markdown-named helpers for YAML-named helpers, which is small but compatibility-relevant for source-checkout automation.

Stored data model
Persistent data-model change detected: migration/backfill/repair: qa/scenarios/runtime/update-run-package-self-upgrade.md, migration/backfill/repair: qa/scenarios/runtime/update-run-package-self-upgrade.yaml, persistent cache schema: src/cli/completion-cli.write-state.test.ts, serialized state: qa/scenarios/agents/subagent-stale-child-links.md, serialized state: qa/scenarios/agents/subagent-stale-child-links.yaml, serialized state: qa/scenarios/runtime/tools/sessions-spawn.yaml, and 5 more. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Wait for the in-progress CI and fresh autoreview rerun to finish before landing.

Risk before merge

  • [P1] The PR intentionally replaces private qa-lab helper exports and the seeded workspace artifact name, so source-checkout QA consumers using the old helper names or QA_SCENARIOS.md need to migrate with this branch.
  • [P1] CI and the fresh autoreview rerun were still in progress in the live PR/body state I inspected, so landing should wait for those maintainer gates rather than treating this review as release proof.

Maintainer options:

  1. Accept the private QA format migration after gates pass (recommended)
    Because qa-lab is private source-checkout tooling, maintainers can intentionally accept the helper/artifact rename once CI and autoreview finish cleanly.
  2. Preserve aliases only if a real caller exists
    If maintainers know of source-checkout automation still importing the old Markdown helper names or reading QA_SCENARIOS.md, add a narrow migration path before merge.

Next step before merge

  • [P2] Protected maintainer PR with no narrow automated repair target; maintainer review should decide when to accept the private QA compatibility migration after gates complete.

Security
Cleared: The diff changes qa-lab parser/tests/docs and YAML fixtures only; I found no concrete supply-chain, workflow, dependency, secret-handling, or permission expansion concern.

Review details

Best possible solution:

Land the direct YAML catalog as the single private QA source format once full qa-lab/build/CI gates pass, without adding runtime shims for the old Markdown scenario helpers unless maintainers identify a real shipped contract.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR is a maintainer refactor of QA scenario source format, not a bug report with a failing user path. The relevant check is whether the catalog loads the converted YAML scenarios and the adjacent tests cover that path.

Is this the best way to solve the issue?

Yes, with maintainer acceptance of the private API rename: replacing fenced Markdown parsing with top-level YAML documents is the cleanest owner-boundary fix for the stated qa-lab catalog cleanup. The safer alternative would be temporary aliases only if maintainers identify real source-checkout consumers of the old helper names.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against d2c0d3ac9bc2.

Label changes

Label changes:

  • add P2: This is a normal-priority maintainer refactor of internal QA infrastructure with limited user runtime blast radius but broad QA workflow impact.
  • add merge-risk: 🚨 compatibility: The PR changes qa-lab exported helper names and the generated QA_SCENARIOS artifact name, so existing private source-checkout consumers may need migration.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The contributor proof gate is not applied because this is a maintainer-labeled PR; the PR body instead lists focused qa-lab tests and build verification for maintainer validation.

Label justifications:

  • P2: This is a normal-priority maintainer refactor of internal QA infrastructure with limited user runtime blast radius but broad QA workflow impact.
  • merge-risk: 🚨 compatibility: The PR changes qa-lab exported helper names and the generated QA_SCENARIOS artifact name, so existing private source-checkout consumers may need migration.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The contributor proof gate is not applied because this is a maintainer-labeled PR; the PR body instead lists focused qa-lab tests and build verification for maintainer validation.
Evidence reviewed

What I checked:

  • Repository policy read: Read the full root AGENTS.md and scoped extensions/docs/test guides; the root review policy applies because this is a maintainer-labeled OpenClaw PR touching bundled plugin QA surfaces and exported helper names. (AGENTS.md:1, d2c0d3ac9bc2)
  • Live PR state: Live GitHub data shows the PR is open, authored by RomneyDa, carries the protected maintainer label, is mergeable but unstable, and has CI still running at the inspected head d50e91b. (d50e91b202b4)
  • Current main still uses Markdown catalog parsing: Current main still points qa-lab at qa/scenarios/index.md and parses fenced qa-pack/qa-scenario/qa-flow blocks, so the PR is not already implemented on main. (extensions/qa-lab/src/scenario-catalog.ts:269, e99ab385cfe1)
  • Proposed parser shape: The PR points the catalog at qa/scenarios/index.yaml, parses top-level pack/scenario/flow YAML documents, enforces flow presence for flow scenarios, and lists only nested .yaml scenario files while excluding index.yaml. (extensions/qa-lab/src/scenario-catalog.ts:283, d50e91b202b4)
  • PR tree reference search: A PR-tree grep shows qa-lab scenario references were updated to index.yaml/QA_SCENARIOS.yaml, with remaining qa-scenario wording limited to evidence kind names and unrelated memory-wiki/docs index.md references. (d50e91b202b4)
  • Adjacent tests cover the new catalog contract: The updated scenario-catalog tests assert YAML path discovery length, required scenario ids, flow loading, coverage metadata, execution config, sourcePath values, and representative live/tool scenario metadata. (extensions/qa-lab/src/scenario-catalog.test.ts:14, d50e91b202b4)

Likely related people:

  • RomneyDa: Dallin Romney has a recent current-main commit touching the qa-lab scenario catalog/corpus area and is also the author of this PR, so routing questions about the YAML conversion to him is supported beyond mere PR authorship. (role: recent area contributor; confidence: high; commits: a3e9dfee0e39, d50e91b202b4; files: extensions/qa-lab/src/scenario-catalog.ts, qa/scenarios)
  • vincentkoc: Git blame attributes the existing Markdown scenario catalog parser and qa/scenarios bootstrap files on current main to Vincent Koc's grafted commit, making him a likely context holder for the current behavior being replaced. (role: current scaffold author; confidence: medium; commits: 9a27af9507b1; files: extensions/qa-lab/src/scenario-catalog.ts, qa/scenarios/index.md)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jun 14, 2026
@RomneyDa
RomneyDa merged commit fef8394 into main Jun 15, 2026
162 checks passed
@RomneyDa
RomneyDa deleted the scenario-yaml branch June 15, 2026 00:31
@clawsweeper clawsweeper Bot mentioned this pull request Jun 15, 2026
Closed
@github-actions github-actions Bot mentioned this pull request Jun 15, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 15, 2026
@RomneyDa
Convert QA scenarios to YAML files (openclaw#92915)
89fc412 
* refactor: load QA scenarios from YAML

* docs: update personal QA scenario docs

* test: keep QA scenarios YAML-only
@clawsweeper clawsweeper Bot mentioned this pull request Jun 15, 2026
Open
@RomneyDa RomneyDa mentioned this pull request Jun 15, 2026
Closed
wangmiao0668000666 pushed a commit to wangmiao0668000666/openclaw that referenced this pull request Jun 17, 2026
@RomneyDa @wangmiao0668000666
Convert QA scenarios to YAML files (openclaw#92915)
c0121fe 
* refactor: load QA scenarios from YAML

* docs: update personal QA scenario docs

* test: keep QA scenarios YAML-only
crh-code pushed a commit to crh-code/openclaw that referenced this pull request Jun 18, 2026
@RomneyDa @crh-code
Convert QA scenarios to YAML files (openclaw#92915)
4314bb8 
* refactor: load QA scenarios from YAML

* docs: update personal QA scenario docs

* test: keep QA scenarios YAML-only
badgerbees pushed a commit to badgerbees/openclaw that referenced this pull request Jul 8, 2026
@RomneyDa @badgerbees
Convert QA scenarios to YAML files (openclaw#92915)
5ac795a 
* refactor: load QA scenarios from YAML

* docs: update personal QA scenario docs

* test: keep QA scenarios YAML-only
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cli CLI command changes docs Improvements or additions to documentation extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa