Skip to content

Commit fef8394

Browse files
RomneyDaRomneyDa
authored
Convert QA scenarios to YAML files (#92915)
* refactor: load QA scenarios from YAML * docs: update personal QA scenario docs * test: keep QA scenarios YAML-only
1 parent 1ca3d4f commit fef8394

267 files changed

Lines changed: 13170 additions & 13504 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/openclaw-qa-testing/SKILL.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Use this skill for `qa-lab` / `qa-channel` work. Repo-local QA only.
1313
- `docs/help/testing.md`
1414
- `docs/channels/qa-channel.md`
1515
- `qa/README.md`
16-
- `qa/scenarios/index.md`
16+
- `qa/scenarios/index.yaml`
1717
- `extensions/qa-lab/src/suite.ts`
1818
- `extensions/qa-lab/src/character-eval.ts`
1919

@@ -198,7 +198,9 @@ pnpm openclaw qa character-eval \
198198
- Judges default to `openai/gpt-5.4,thinking=xhigh,fast` and `anthropic/claude-opus-4-6,thinking=high`.
199199
- Report includes judge ranking, run stats, durations, and full transcripts; do not include raw judge replies. Duration is benchmark context, not a grading signal.
200200
- Candidate and judge concurrency default to 16. Use `--concurrency <n>` and `--judge-concurrency <n>` to override when local gateways or provider limits need a gentler lane.
201-
- Scenario source should stay markdown-driven under `qa/scenarios/`.
201+
- Scenario source is YAML-only under `qa/scenarios/`: use `index.yaml` and
202+
per-scenario `*.yaml` files with top-level `title`, `scenario`, and optional
203+
`flow`. Never add fenced `qa-scenario` / `qa-flow` Markdown files.
202204
- For isolated character/persona evals, write the persona into `SOUL.md` and blank `IDENTITY.md` in the scenario flow. Use `SOUL.md + IDENTITY.md` only when intentionally testing how the normal OpenClaw identity combines with the character.
203205
- Keep prompts natural and task-shaped. The candidate model should receive character setup through `SOUL.md`, then normal user turns such as chat, workspace help, and small file tasks; do not ask "how would you react?" or tell the model it is in an eval.
204206
- Prefer at least one real task, such as creating or editing a tiny workspace artifact, so the transcript captures character under normal tool use instead of pure roleplay.
@@ -234,7 +236,8 @@ pnpm openclaw qa manual \
234236

235237
## Repo facts
236238

237-
- Seed scenarios live in `qa/`.
239+
- Seed scenarios live in `qa/scenarios/index.yaml` and
240+
`qa/scenarios/<theme>/*.yaml`.
238241
- Main live runner: `extensions/qa-lab/src/suite.ts`
239242
- QA lab server: `extensions/qa-lab/src/lab-server.ts`
240243
- Child gateway harness: `extensions/qa-lab/src/gateway-child.ts`
@@ -262,8 +265,9 @@ pnpm openclaw qa manual \
262265

263266
## When adding scenarios
264267

265-
- Add or update scenario markdown under `qa/scenarios/`
266-
- Keep kickoff expectations in `qa/scenarios/index.md` aligned
268+
- Add or update scenario YAML under `qa/scenarios/`; do not add `.md` scenario
269+
files or fenced YAML blocks.
270+
- Keep kickoff expectations in `qa/scenarios/index.yaml` aligned
267271
- Add executable coverage in `extensions/qa-lab/src/suite.ts`
268272
- Prefer end-to-end assertions over mock-only checks
269273
- Save outputs under `.artifacts/qa-e2e/`

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,7 @@ Skills own workflows; root owns hard policy and routing.
214214

215215
- Vitest. Colocated `*.test.ts`; e2e `*.e2e.test.ts`; example models `sonnet-4.6`, `gpt-5.5`; test GPT with 5.5 preferred, 5.4 ok; no GPT-4.x agent-smoke defaults.
216216
- Prefer behavior tests over workflow/docs string greps. Put operator policy reminders in AGENTS/docs.
217+
- QA scenario sources are YAML only: `qa/scenarios/index.yaml` and `qa/scenarios/<theme>/*.yaml`. Do not add fenced `qa-scenario`/`qa-flow` Markdown files under `qa/scenarios/`.
217218
- Clean timers/env/globals/mocks/sockets/temp dirs/module state; `--isolate=false` safe.
218219
- Prefer injection and narrow `*.runtime.ts` mocks over broad barrels or `openclaw/plugin-sdk/*`.
219220
- Do not edit baseline/inventory/ignore/snapshot/expected-failure files to silence checks without explicit approval.

‎docs/concepts/personal-agent-benchmark-pack.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ The Personal Agent Benchmark Pack is a small repo-backed QA scenario pack for
1111
local personal assistant workflows. It is not a generic model benchmark and it
1212
does not require a new runner. The pack reuses the private QA stack described in
1313
[QA overview](/concepts/qa-e2e-automation), the synthetic
14-
[QA channel](/channels/qa-channel), and the existing `qa/scenarios` markdown
14+
[QA channel](/channels/qa-channel), and the existing `qa/scenarios` YAML
1515
catalog.
1616

1717
The first pack is intentionally narrow:
@@ -61,9 +61,9 @@ to inspect and file in issues.
6161

6262
## Extending The Pack
6363

64-
Add new cases under `qa/scenarios/personal/`, then add the scenario id to
65-
`QA_PERSONAL_AGENT_SCENARIO_IDS`. Keep each case small, local, deterministic in
66-
`mock-openai`, and focused on one personal assistant behavior.
64+
Add new `.yaml` cases under `qa/scenarios/personal/`, then add the scenario id
65+
to `QA_PERSONAL_AGENT_SCENARIO_IDS`. Keep each case small, local, deterministic
66+
in `mock-openai`, and focused on one personal assistant behavior.
6767

6868
Good follow-up candidates:
6969

‎docs/concepts/qa-e2e-automation.md‎

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ script aliases; both forms are supported.
3333
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
3434
| `qa run` | Bundled QA self-check; writes a Markdown report. |
3535
| `qa suite` | Run repo-backed scenarios against the QA gateway lane. Aliases: `pnpm openclaw qa suite --runner multipass` for a disposable Linux VM. |
36-
| `qa coverage` | Print the markdown scenario-coverage inventory (`--json` for machine output). |
36+
| `qa coverage` | Print the YAML scenario-coverage inventory (`--json` for machine output). |
3737
| `qa parity-report` | Compare two `qa-suite-summary.json` files and write the agentic parity report, or use `--runtime-axis --token-efficiency` to write Codex-vs-OpenClaw runtime parity and token-efficiency reports from one runtime-pair summary. |
3838
| `qa character-eval` | Run the character QA scenario across multiple live models with a judged report. See [Reporting](#reporting). |
3939
| `qa manual` | Run a one-off prompt against the selected provider/model lane. |
@@ -769,25 +769,26 @@ Operational env vars and the Convex broker endpoint contract live in [Testing
769769

770770
Seed assets live in `qa/`:
771771

772-
- `qa/scenarios/index.md`
773-
- `qa/scenarios/<theme>/*.md`
772+
- `qa/scenarios/index.yaml`
773+
- `qa/scenarios/<theme>/*.yaml`
774774

775775
These are intentionally in git so the QA plan is visible to both humans and the
776776
agent.
777777

778-
`qa-lab` should stay a generic markdown runner. Each scenario markdown file is
778+
`qa-lab` should stay a generic YAML scenario runner. Each scenario YAML file is
779779
the source of truth for one test run and should define:
780780

781-
- scenario metadata
782-
- optional category, capability, lane, and risk metadata
783-
- docs and code refs
784-
- optional plugin requirements
785-
- optional gateway config patch
786-
- an executable `qa-flow` block for flow scenarios, or `execution.kind`/`execution.path`
787-
for Vitest and Playwright scenarios
788-
789-
The reusable runtime surface that backs `qa-flow` blocks is allowed to stay generic
790-
and cross-cutting. For example, markdown scenarios can combine transport-side
781+
- top-level `title`
782+
- `scenario` metadata
783+
- optional category, capability, lane, and risk metadata in `scenario`
784+
- docs and code refs in `scenario`
785+
- optional plugin requirements in `scenario`
786+
- optional gateway config patch in `scenario`
787+
- executable top-level `flow` for flow scenarios, or `scenario.execution.kind` /
788+
`scenario.execution.path` for Vitest and Playwright scenarios
789+
790+
The reusable runtime surface that backs `flow` is allowed to stay generic
791+
and cross-cutting. For example, YAML scenarios can combine transport-side
791792
helpers with browser-side helpers that drive the embedded Control UI through the
792793
Gateway `browser.request` seam without adding a special-case runner.
793794

@@ -825,17 +826,17 @@ provider names.
825826

826827
## Transport adapters
827828

828-
`qa-lab` owns a generic transport seam for markdown QA scenarios. `qa-channel` is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.
829+
`qa-lab` owns a generic transport seam for YAML QA scenarios. `qa-channel` is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.
829830

830831
At the architecture level, the split is:
831832

832833
- `qa-lab` owns generic scenario execution, worker concurrency, artifact writing, and reporting.
833834
- The transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
834-
- Markdown scenario files under `qa/scenarios/` define the test run; `qa-lab` provides the reusable runtime surface that executes them.
835+
- YAML scenario files under `qa/scenarios/` define the test run; `qa-lab` provides the reusable runtime surface that executes them.
835836

836837
### Adding a channel
837838

838-
Adding a channel to the markdown QA system requires exactly two things:
839+
Adding a channel to the YAML QA system requires exactly two things:
839840

840841
1. A transport adapter for the channel.
841842
2. A scenario pack that exercises the channel contract.
@@ -869,7 +870,7 @@ The minimum adoption bar for a new channel:
869870
2. Implement the transport runner on the shared `qa-lab` host seam.
870871
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
871872
4. Mount the runner as `openclaw qa <runner>` instead of registering a competing root command. Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`. Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints.
872-
5. Author or adapt markdown scenarios under the themed `qa/scenarios/` directories.
873+
5. Author or adapt YAML scenarios under the themed `qa/scenarios/` directories.
873874
6. Use the generic scenario helpers for new scenarios.
874875
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
875876

‎extensions/qa-lab/api.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ export {
6464
export {
6565
DEFAULT_QA_AGENT_IDENTITY_MARKDOWN,
6666
hasQaScenarioPack,
67-
listQaScenarioMarkdownPaths,
67+
listQaScenarioYamlPaths,
6868
type QaBootstrapScenarioCatalog,
6969
type QaScenarioExecution,
7070
type QaScenarioFlow,
@@ -76,7 +76,7 @@ export {
7676
readQaScenarioExecutionConfig,
7777
readQaScenarioOverviewMarkdown,
7878
readQaScenarioPack,
79-
readQaScenarioPackMarkdown,
79+
readQaScenarioPackYamlSource,
8080
validateQaScenarioExecutionConfig,
8181
} from "./src/scenario-catalog.js";
8282
export { createQaSelfCheckScenario } from "./src/self-check-scenario.js";

‎extensions/qa-lab/src/cli.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -436,7 +436,7 @@ export function registerQaLabCli(program: Command) {
436436
);
437437

438438
qa.command("coverage")
439-
.description("Print the markdown QA coverage inventory")
439+
.description("Print the YAML QA coverage inventory")
440440
.option("--repo-root <path>", "Repository root to target when writing --output")
441441
.option("--output <path>", "Write the coverage inventory to this path")
442442
.option("--json", "Print JSON instead of Markdown", false)

‎extensions/qa-lab/src/coverage-report.test.ts‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ function scenarioWithCoverage(params: {
9292
},
9393
objective: "Exercise test coverage.",
9494
successCriteria: ["Evidence is recorded."],
95-
sourcePath: params.sourcePath ?? "qa/scenarios/test/test-scenario.md",
95+
sourcePath: params.sourcePath ?? "qa/scenarios/test/test-scenario.yaml",
9696
execution,
9797
};
9898
}
@@ -157,7 +157,7 @@ describe("qa coverage report", () => {
157157
kind: "playwright",
158158
path: "ui/src/ui/e2e/chat-flow.e2e.test.ts",
159159
role: "primary",
160-
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.md"],
160+
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.yaml"],
161161
});
162162
expect(inventory.scenarioPacks.map((pack) => pack.id)).toEqual([
163163
"observability",
@@ -184,7 +184,7 @@ describe("qa coverage report", () => {
184184
expect(report).toContain("- Missing coverage metadata: 0");
185185
expect(report).toContain("- Overlapping coverage IDs:");
186186
expect(report).toContain("memory.recall");
187-
expect(report).toContain("primary: memory-recall (qa/scenarios/memory/memory-recall.md)");
187+
expect(report).toContain("primary: memory-recall (qa/scenarios/memory/memory-recall.yaml)");
188188
expect(report).toContain("secondary: active-memory-preprompt-recall");
189189
expect(report).toContain("## Scenario Packs");
190190
expect(report).toContain(
@@ -236,7 +236,7 @@ describe("qa coverage report", () => {
236236
primary: [TEST_BROWSER_COVERAGE_ID],
237237
executionKind: "playwright",
238238
executionPath: playwrightExecutionPath,
239-
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.md",
239+
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.yaml",
240240
});
241241
const report = renderQaScenarioMatchesMarkdownReport({
242242
query: "mixed",
@@ -310,7 +310,7 @@ describe("qa coverage report", () => {
310310
scenarios: [
311311
scenarioWithCoverage({
312312
primary: [TEST_BROWSER_COVERAGE_ID],
313-
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.md",
313+
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.yaml",
314314
executionKind: "playwright",
315315
executionPath: "ui/src/ui/e2e/chat-flow.e2e.test.ts",
316316
}),
@@ -322,15 +322,15 @@ describe("qa coverage report", () => {
322322
expect(report.fulfilledFeatureCount).toBe(1);
323323
expect(report.categories[0]?.mappingStatus).toBe("mapped");
324324
expect(report.categories[0]?.scenarioRefs).toStrictEqual([
325-
"qa/scenarios/ui/control-ui-chat-flow-playwright.md",
325+
"qa/scenarios/ui/control-ui-chat-flow-playwright.yaml",
326326
]);
327327
expect(report.categories[0]?.evidence).toStrictEqual([
328328
{
329329
coverageId: TEST_BROWSER_COVERAGE_ID,
330330
kind: "playwright",
331331
path: "ui/src/ui/e2e/chat-flow.e2e.test.ts",
332332
role: "primary",
333-
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.md"],
333+
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.yaml"],
334334
},
335335
]);
336336
});
@@ -389,22 +389,22 @@ describe("qa coverage report", () => {
389389
scenarios: [
390390
scenarioWithCoverage({
391391
primary: [TEST_EXECUTABLE_COVERAGE_ID],
392-
sourcePath: "qa/scenarios/channels/dm-chat-baseline.md",
392+
sourcePath: "qa/scenarios/channels/dm-chat-baseline.yaml",
393393
}),
394394
],
395395
});
396396

397397
expect(report.validationIssues).toStrictEqual([]);
398398
expect(report.categories[0]?.scenarioRefs).toStrictEqual([
399-
"qa/scenarios/channels/dm-chat-baseline.md",
399+
"qa/scenarios/channels/dm-chat-baseline.yaml",
400400
]);
401401
expect(report.categories[0]?.evidence).toStrictEqual([
402402
{
403403
coverageId: TEST_EXECUTABLE_COVERAGE_ID,
404404
kind: "qa-scenario",
405405
path: null,
406406
role: "primary",
407-
scenarioRefs: ["qa/scenarios/channels/dm-chat-baseline.md"],
407+
scenarioRefs: ["qa/scenarios/channels/dm-chat-baseline.yaml"],
408408
},
409409
]);
410410
});

‎extensions/qa-lab/src/coverage-report.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -505,7 +505,7 @@ export function renderQaScenarioMatchesMarkdownReport(params: {
505505
lines.push(` - surface: ${match.surfaces.join(", ")}`);
506506
lines.push(
507507
match.executionKind === "flow"
508-
? " - execution: qa-flow"
508+
? " - execution: flow"
509509
: ` - execution: ${match.executionKind} ${match.executionPath ?? "missing"}`,
510510
);
511511
lines.push(` - coverage IDs: ${match.coverageIds.join(", ") || "none"}`);

‎extensions/qa-lab/src/discovery-eval.test.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ describe("qa discovery evaluation", () => {
1010
it("accepts rich discovery reports that explicitly confirm all required files were read", () => {
1111
const report = `
1212
Worked
13-
- Read all three requested files: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, and repo/docs/help/testing.md.
13+
- Read all three requested files: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, and repo/docs/help/testing.md.
1414
Failed
1515
- None.
1616
Blocked
@@ -29,7 +29,7 @@ The helper text mentions banned phrases like "not present", "missing files", "bl
2929
it("accepts numeric 'all 4 required files read' confirmations", () => {
3030
const report = `
3131
Worked
32-
- Source: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md
32+
- Source: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md
3333
- all 3 required files read.
3434
Failed
3535
- None.
@@ -50,7 +50,7 @@ The report may quote phrases like "not present" while describing the evaluator,
5050
const report = `
5151
Worked
5252
- All three files retrieved. Now let me compile the protocol report.
53-
- All three mandated files read successfully: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
53+
- All three mandated files read successfully: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
5454
Failed
5555
- None.
5656
Blocked
@@ -84,7 +84,7 @@ Follow-up
8484
it("flags discovery replies that drift into unrelated suite wrap-up claims", () => {
8585
const report = `
8686
Worked
87-
- All three requested files were read: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
87+
- All three requested files were read: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
8888
Failed
8989
- None.
9090
Blocked

‎extensions/qa-lab/src/discovery-eval.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ function readRequiredDiscoveryRefs() {
88
| undefined;
99
return (
1010
config?.requiredFiles ?? [
11-
"repo/qa/scenarios/index.md",
11+
"repo/qa/scenarios/index.yaml",
1212
"repo/extensions/qa-lab/src/suite.ts",
1313
"repo/docs/help/testing.md",
1414
]

0 commit comments

Comments
 (0)