Skip to content

test(qa): QA smoke profile fixes and performance improvements#96340

Merged
RomneyDa merged 1 commit into
mainfrom
codex/qa-smoke-speed-fixes
Jun 24, 2026
Merged

test(qa): QA smoke profile fixes and performance improvements#96340
RomneyDa merged 1 commit into
mainfrom
codex/qa-smoke-speed-fixes

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jun 24, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

The QA smoke CI profile was exceeding the intended walltime after adding the full smoke profile shard. The slowest cases were not basic smoke coverage: two live Claude CLI provider checks were running in the mock profile, one MCP check booted a full gateway/memory flow for a direct MCP contract, and native test-file proof waited behind isolated flow work.

Why This Change Was Made

  • Marks the two Claude CLI capability scenarios as live-frontier only so mock smoke-ci excludes them through the existing taxonomy/profile filtering path.
  • Converts mcp-plugin-tools-call from a full QA flow to a direct Vitest proof using the real MCP SDK client and in-memory transport against createPluginToolsMcpServer.
  • Starts shared flow partitions first, then native test-file partitions, then isolated flow partitions so the weighted queue does not starve cheap native proof behind long isolated work.
  • Fixes the mock OpenAI memory planner to continue memory tool planning from path-only memory_search results, and removes the smoke scenario's over-specific memory_get assertion from memory-tools-channel-context.

User Impact

The smoke CI profile keeps the same broad smoke coverage intent while avoiding live-provider checks and heavyweight flow setup that do not belong in mock smoke. Local full-profile smoke now finishes in about 3 minutes on this checkout instead of the prior CI run's 8m42s failure tail.

Evidence

  • node scripts/run-vitest.mjs extensions/qa-lab/src/providers/mock-openai/server.test.ts extensions/qa-lab/src/suite-launch.runtime.test.ts extensions/qa-lab/src/cli.runtime.test.ts src/mcp/plugin-tools-mcp-client.test.ts -> 1 unit test file and 3 extension QA test files passed; 206 tests total.
  • node scripts/run-vitest.mjs extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/test-file-scenario-runner.test.ts extensions/qa-lab/src/coverage-report.test.ts -> 3 extension QA test files passed; 58 tests total.
  • ./node_modules/.bin/oxfmt --check --threads=1 extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/providers/mock-openai/server.test.ts extensions/qa-lab/src/providers/mock-openai/server.ts extensions/qa-lab/src/suite-launch.runtime.test.ts extensions/qa-lab/src/suite-launch.runtime.ts src/mcp/plugin-tools-mcp-client.test.ts -> clean.
  • node scripts/run-oxlint.mjs extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/providers/mock-openai/server.test.ts extensions/qa-lab/src/providers/mock-openai/server.ts extensions/qa-lab/src/suite-launch.runtime.test.ts extensions/qa-lab/src/suite-launch.runtime.ts src/mcp/plugin-tools-mcp-client.test.ts -> clean.
  • git diff --check -> clean.
  • .agents/skills/autoreview/scripts/autoreview --mode local ... -> clean; no accepted/actionable findings.
  • Full smoke profile: OPENCLAW_QA_SUITE_PROGRESS=1 OPENCLAW_BUILD_PRIVATE_QA=1 OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 OPENCLAW_DISABLE_BUNDLED_PLUGINS=0 OPENCLAW_QA_REDACT_PUBLIC_METADATA=1 OPENCLAW_QA_TRANSPORT_READY_TIMEOUT_MS=180000 NODE_OPTIONS=--max-old-space-size=8192 node scripts/run-node.mjs qa run --repo-root . --qa-profile smoke-ci --concurrency 8 --output-dir .artifacts/qa-e2e/smoke-ci-speed-full-4 -> 78/78 passed; shell walltime 2:51.97, QA summary window 2:42.694.

@clawsweeper

clawsweeper Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 24, 2026, 4:57 AM ET / 08:57 UTC.

Summary
The branch narrows QA smoke scenario selection, reorders unified suite partitions, replaces one MCP flow scenario with direct Vitest MCP-client coverage, and adjusts mock OpenAI memory-tool planning.

PR surface: Source +9, Tests +196, Other -41. Total +164 across 10 files.

Reproducibility: Source-reproducible, not locally rerun: current main appends all flow partitions before native test-file tasks, and smoke-ci defaults to mock-openai while provider-mode filtering depends on scenario metadata. The PR body supplies the runtime timing evidence for the walltime improvement.

Review metrics: 1 noteworthy metric.

  • Smoke Coverage Reshaped: 2 provider gates added; 1 flow scenario converted to Vitest; 1 mock assertion relaxed. These are the coverage changes maintainers should consciously accept because the PR optimizes walltime by moving checks out of the mock smoke path.

Stored data model
Persistent data-model change detected: serialized state: extensions/qa-lab/src/providers/mock-openai/server.test.ts, serialized state: qa/scenarios/plugins/mcp-plugin-tools-call.yaml, vector/embedding metadata: qa/scenarios/memory/memory-tools-channel-context.yaml. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Explicitly accept the MCP full-flow-to-Vitest smoke coverage tradeoff before merge.

Risk before merge

  • [P1] The PR intentionally speeds smoke by moving one MCP check out of a full gateway/memory flow and relaxing one mock-only memory_get assertion; maintainers should be comfortable that direct MCP-client coverage plus remaining memory coverage is enough for smoke-ci.

Maintainer options:

  1. Accept Faster Smoke Coverage (recommended)
    Proceed if maintainers agree that direct MCP bridge proof and the remaining memory scenarios preserve the intended smoke signal while restoring walltime.
  2. Keep Heavy MCP Flow Elsewhere
    Before merge, add or point to a release/all-profile flow that retains gateway-backed MCP memory coverage outside smoke-ci.

Next step before merge

  • No automated repair is needed; this maintainer-labeled PR should stay in the normal maintainer merge flow with the smoke coverage tradeoff visible.

Security
Cleared: No concrete security or supply-chain concern found; the diff changes QA tests/scenario metadata and does not add dependencies, workflow permissions, scripts, or package-resolution changes.

Review details

Best possible solution:

A faster smoke profile that uses existing provider-mode gating and direct MCP bridge proof, with heavier end-to-end MCP or memory coverage left to non-smoke lanes if maintainers want that depth.

Do we have a high-confidence way to reproduce the issue?

Source-reproducible, not locally rerun: current main appends all flow partitions before native test-file tasks, and smoke-ci defaults to mock-openai while provider-mode filtering depends on scenario metadata. The PR body supplies the runtime timing evidence for the walltime improvement.

Is this the best way to solve the issue?

Yes, with a maintainer-owned coverage tradeoff: using existing provider-mode metadata and task ordering is the narrowest maintainable fix. The only safer alternative is to keep the heavier MCP gateway-memory flow in a non-smoke lane if maintainers want that coverage depth.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f65aca64fc37.

Label changes

Label changes:

  • add P2: The PR addresses a concrete QA smoke walltime regression with limited maintainer-facing blast radius.
  • add merge-risk: 🚨 automation: The diff changes QA smoke automation semantics by excluding live-provider checks from mock smoke and replacing one full flow with direct Vitest coverage.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Maintainer-authored PR, so the external-contributor real behavior proof gate does not apply; the PR body nevertheless includes focused checks and a full smoke-ci 78/78 timing run.

Label justifications:

  • P2: The PR addresses a concrete QA smoke walltime regression with limited maintainer-facing blast radius.
  • merge-risk: 🚨 automation: The diff changes QA smoke automation semantics by excluding live-provider checks from mock smoke and replacing one full flow with direct Vitest coverage.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Maintainer-authored PR, so the external-contributor real behavior proof gate does not apply; the PR body nevertheless includes focused checks and a full smoke-ci 78/78 timing run.
Evidence reviewed

PR surface:

Source +9, Tests +196, Other -41. Total +164 across 10 files.

View PR surface stats
Area Files Added Removed Net
Source 2 19 10 +9
Tests 4 196 0 +196
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 4 8 49 -41
Total 10 223 59 +164

What I checked:

  • Repository policy read: Root AGENTS.md and extensions/AGENTS.md were read fully; their full-surface review, dependency inspection, plugin-boundary, and protected-label guidance shaped the review. (AGENTS.md:1, f65aca64fc37)
  • Live PR state: Live GitHub data shows this PR is open, mergeable, maintainer-labeled, authored by a repository member, and has no review comments; the protected label prevents cleanup close. (953c9d8878aa)
  • Current smoke profile filtering path: Current main defaults smoke-ci to mock-openai and filters selected taxonomy scenarios through scenarioMatchesQaProviderLane, which honors execution.config.requiredProviderMode when scenario metadata declares it. (extensions/qa-lab/src/cli.runtime.ts:732, f65aca64fc37)
  • Current weighted queue behavior: Current main builds unified suite tasks with all flow partitions before the single test-file partition, so an isolated flow partition can occupy the weighted queue before native test-file proof starts. (extensions/qa-lab/src/suite-launch.runtime.ts:451, f65aca64fc37)
  • MCP bridge contract checked: The existing MCP bridge creates a Server from injected tools, maps listTools/callTool through plugin-tool handlers, and the pinned MCP SDK Client exposes listTools/callTool over InMemoryTransport, matching the new direct MCP-client test shape. (src/mcp/plugin-tools-serve.ts:57, f65aca64fc37)
  • PR validation evidence: The PR body reports focused Vitest/QA test commands, format/lint checks, local autoreview, and a full smoke-ci run with 78/78 scenarios passing in 2:51.97 walltime. (953c9d8878aa)

Likely related people:

  • vincentkoc: Current-main blame and log for the QA suite launcher, mock OpenAI server, and MCP plugin-tools files point to recent commits by Vincent Koc, including the adjacent merged mock-provider fix on the same day. (role: recent area contributor; confidence: medium; commits: 4d034639adfd, f65aca64fc37; files: extensions/qa-lab/src/suite-launch.runtime.ts, extensions/qa-lab/src/providers/mock-openai/server.ts, src/mcp/plugin-tools-serve.ts)
  • joshavant: Merged PR fix: forward MCP tool abort signals #82443 changed the MCP plugin-tools handler/test path that this PR now exercises with a real MCP client. (role: adjacent MCP contributor; confidence: medium; commits: c807771f4c5d, 5fd8d25e6e13, b7d61c8daffc; files: src/mcp/plugin-tools-handlers.ts, src/mcp/plugin-tools-handlers.cancel.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jun 24, 2026
@RomneyDa RomneyDa changed the title Speed up QA smoke profile test(qa): QA smoke profile fixes and performance improvements Jun 24, 2026
@RomneyDa
RomneyDa merged commit f29dbd3 into main Jun 24, 2026
150 of 158 checks passed
@RomneyDa
RomneyDa deleted the codex/qa-smoke-speed-fixes branch June 24, 2026 16:31
@github-actions github-actions Bot mentioned this pull request Jun 24, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 25, 2026
@RomneyDa
test(qa): speed up smoke profile (openclaw#96340)
267cd58
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
@RomneyDa @QiuYuang
test(qa): speed up smoke profile (openclaw#96340)
e4b3b58
@clawsweeper clawsweeper Bot mentioned this pull request Jul 3, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa