Skip to content

Commit d8a1a7d

Browse files
RomneyDaRomneyDa
committed
test(qa): preserve root profile parsing
1 parent a78fb0c commit d8a1a7d

4 files changed

Lines changed: 113 additions & 7 deletions

File tree

‎docs/concepts/qa-e2e-automation.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ script aliases; both forms are supported.
3131

3232
| Command | Purpose |
3333
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
34-
| `qa run` | Bundled QA self-check; writes a Markdown report. |
34+
| `qa run` | Bundled QA self-check without `--profile`; mapped maturity profile runner with `--profile smoke-ci` or `--profile release`. |
3535
| `qa suite` | Run repo-backed scenarios against the QA gateway lane. Aliases: `pnpm openclaw qa suite --runner multipass` for a disposable Linux VM. |
3636
| `qa coverage` | Print the markdown scenario-coverage inventory (`--json` for machine output). |
3737
| `qa parity-report` | Compare two `qa-suite-summary.json` files and write the agentic parity report, or use `--runtime-axis --token-efficiency` to write Codex-vs-OpenClaw runtime parity and token-efficiency reports from one runtime-pair summary. |
@@ -51,6 +51,26 @@ script aliases; both forms are supported.
5151
| `qa whatsapp` | Live transport lane against real WhatsApp Web accounts. |
5252
| `qa mantis` | Before and after verification runner for live transport bugs, with Discord status-reactions evidence, Crabbox desktop/browser smoke, and Slack-in-VNC smoke. See [Mantis](/concepts/mantis) and [Mantis Slack Desktop Runbook](/concepts/mantis-slack-desktop-runbook). |
5353

54+
Mapped `qa run` profiles read membership from `taxonomy-mappings.yaml`, then
55+
dispatch the resolved scenarios through `qa suite`. `--surface` and
56+
`--category` filter the selected profile instead of defining separate lanes:
57+
58+
```bash
59+
pnpm openclaw qa run \
60+
--profile smoke-ci \
61+
--category agent-runtime-and-provider-execution.agent-turn-execution \
62+
--provider-mode mock-openai \
63+
--output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch
64+
```
65+
66+
Use `smoke-ci` for deterministic no-live-service proof and `release` for the
67+
Stable/LTS proof lane. When a command also needs an OpenClaw root profile, put
68+
the root profile before the QA command:
69+
70+
```bash
71+
pnpm openclaw --profile work qa run --profile smoke-ci
72+
```
73+
5474
## Operator flow
5575

5676
The current QA operator flow is a two-pane QA site:

‎qa/AGENTS.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Scope: `qa/**`.
1616
## QA Scenarios
1717

1818
- Scenario files live under `qa/scenarios/**` and are consumed by QA Lab.
19-
- Keep coverage metadata stable. Use `coverage.primary` for the scorecard category's main coverage IDs and `coverage.secondary` for supporting proof.
19+
- Keep coverage metadata stable. Use `coverage.primary` for the mapped category's main coverage IDs and `coverage.secondary` for supporting proof.
2020
- `docsRefs`, `codeRefs`, and `scenarioRefs` are repo-root relative paths.
2121
- When adding a scenario for maturity proof, update `taxonomy-mappings.yaml` in the same change or leave the scorecard gap explicit.
2222
- Do not put secrets, raw credentials, phone numbers, or unredacted transcripts in scenarios or evidence artifacts.
@@ -33,7 +33,9 @@ Scope: `qa/**`.
3333

3434
- `qa suite` writes `qa-suite-summary.json`; mapped `qa run` dispatches through `qa suite`.
3535
- Normalized summaries include an `evidence` block with `kind`, `schemaVersion`, `generatedAt`, and `entries`.
36-
- Evidence entries carry scenario IDs, coverage IDs, source/docs/code refs, scorecard surface/category IDs, profile, provider/model live state, channel/driver live state, runner, package source, environment, artifact paths, status, failure, and timing.
36+
- Evidence entries are grouped as `test`, `mapping`, `execution`, and `result`.
37+
- `mapping.coverage[]` carries coverage IDs, surface/category IDs, relation role, and ref IDs; `mapping.refs[]` carries docs/code paths once.
38+
- `execution` carries provider/model, channel, runner, package source, environment, and artifacts; `result` carries status, failure, and timing.
3739
- Evidence summaries do not copy taxonomy provenance. Join evidence to maturity state through `taxonomy-mappings.yaml` and `taxonomy.yaml`.
3840

3941
## Validation

‎src/cli/profile.test.ts‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,66 @@ describe("parseCliProfileArgs", () => {
138138
]);
139139
});
140140

141+
it("preserves qa run --profile=release for the command parser", () => {
142+
const res = parseCliProfileArgs([
143+
"node",
144+
"openclaw",
145+
"qa",
146+
"run",
147+
"--profile=release",
148+
"--surface",
149+
"agent-runtime-and-provider-execution",
150+
]);
151+
if (!res.ok) {
152+
throw new Error(res.error);
153+
}
154+
expect(res.profile).toBeNull();
155+
expect(res.argv).toEqual([
156+
"node",
157+
"openclaw",
158+
"qa",
159+
"run",
160+
"--profile=release",
161+
"--surface",
162+
"agent-runtime-and-provider-execution",
163+
]);
164+
});
165+
166+
it("keeps non-mapped qa run --profile values as root profiles", () => {
167+
const res = parseCliProfileArgs([
168+
"node",
169+
"openclaw",
170+
"qa",
171+
"run",
172+
"--profile",
173+
"work",
174+
"--output",
175+
"qa-report.md",
176+
]);
177+
if (!res.ok) {
178+
throw new Error(res.error);
179+
}
180+
expect(res.profile).toBe("work");
181+
expect(res.argv).toEqual(["node", "openclaw", "qa", "run", "--output", "qa-report.md"]);
182+
});
183+
184+
it("keeps non-mapped qa run --profile= values as root profiles", () => {
185+
const res = parseCliProfileArgs([
186+
"node",
187+
"openclaw",
188+
"qa",
189+
"run",
190+
"--profile=work",
191+
"--output",
192+
"qa-report.md",
193+
]);
194+
if (!res.ok) {
195+
throw new Error(res.error);
196+
}
197+
expect(res.profile).toBe("work");
198+
expect(res.argv).toEqual(["node", "openclaw", "qa", "run", "--output", "qa-report.md"]);
199+
});
200+
141201
it("still parses root --profile before Matrix QA", () => {
142202
const res = parseCliProfileArgs([
143203
"node",

‎src/cli/profile.ts‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,33 @@ type CliProfileParseResult =
1616
| { ok: true; profile: string | null; argv: string[] }
1717
| { ok: false; error: string };
1818

19-
function isCommandLocalProfileOption(out: string[]): boolean {
20-
const [primary, secondary] = resolveCliArgvInvocation(out).commandPath;
21-
return primary === "qa" && (secondary === "matrix" || secondary === "run");
19+
const QA_RUN_COMMAND_LOCAL_PROFILE_IDS = new Set(["smoke-ci", "release"]);
20+
21+
function isQaRunCommandLocalProfileValue(value: string | undefined): boolean {
22+
return value ? QA_RUN_COMMAND_LOCAL_PROFILE_IDS.has(value.trim()) : false;
23+
}
24+
25+
function isCommandLocalProfileOption(params: {
26+
arg: string;
27+
args: readonly string[];
28+
index: number;
29+
out: string[];
30+
}): boolean {
31+
const [primary, secondary] = resolveCliArgvInvocation(params.out).commandPath;
32+
if (primary !== "qa") {
33+
return false;
34+
}
35+
if (secondary === "matrix") {
36+
return true;
37+
}
38+
if (secondary !== "run") {
39+
return false;
40+
}
41+
if (params.arg.startsWith("--profile=")) {
42+
return isQaRunCommandLocalProfileValue(params.arg.slice("--profile=".length));
43+
}
44+
const next = params.args[params.index + 1];
45+
return !isValueToken(next) || isQaRunCommandLocalProfileValue(next);
2246
}
2347

2448
export function parseCliProfileArgs(argv: string[]): CliProfileParseResult {
@@ -41,7 +65,7 @@ export function parseCliProfileArgs(argv: string[]): CliProfileParseResult {
4165
}
4266

4367
if (arg === "--profile" || arg.startsWith("--profile=")) {
44-
if (isCommandLocalProfileOption(out)) {
68+
if (isCommandLocalProfileOption({ arg, args, index, out })) {
4569
out.push(arg);
4670
if (arg === "--profile" && isValueToken(args[index + 1])) {
4771
out.push(args[index + 1]);

0 commit comments

Comments
 (0)