QA: fix private runtime source loading#67428
Conversation
🔒 Aisle Security AnalysisWe found 4 potential security issue(s) in this PR:
1. 🟠 Arbitrary code execution via AsyncFunction expression evaluation with exposed qaImport dynamic import
Description
If QA scenario markdown/YAML can be modified by an untrusted party (e.g., a PR author, a downloaded scenario pack, or any user-supplied scenario content), this becomes straightforward arbitrary code execution:
Vulnerable code (new behavior enabling arbitrary imports): function createEvalContext(api: QaFlowApi, vars: QaFlowVars) {
return {
...api,
qaImport: (specifier: string) => import(specifier),
vars,
...vars,
};
}Because (await qaImport('node:child_process')).execSync('...')This is effectively unsandboxed code execution within the process running QA flows. RecommendationTreat scenario Options (pick based on needed functionality):
const ALLOWED_IMPORTS = new Set(["node:path", "node:url"]);
function qaImport(specifier: string) {
if (!ALLOWED_IMPORTS.has(specifier)) {
throw new Error(`Import not allowed: ${specifier}`);
}
return import(specifier);
}
At minimum, document and enforce that scenario flow files are trusted-only, and prevent execution of scenarios from untrusted sources in CI/CD contexts. 2. 🟠 Untrusted search path for Node resolution allows PATH hijacking (executing attacker-controlled node binary)
Description
Vulnerable code: const locator = platform === "win32" ? "where" : "which";
({ stdout } = await execFileImpl(locator, ["node"], {
encoding: "utf8",
env: params?.env,
}));
...
return resolved;If an attacker can influence RecommendationAvoid resolving the executable via untrusted Options (choose one appropriate to the product’s distribution model):
Example (explicit config + validation): import { access } from "node:fs/promises";
import { constants } from "node:fs";
export async function resolveQaNodeExecPath(): Promise<string> {
if (process.versions.bun !== undefined) {
const configured = process.env.OPENCLAW_QA_NODE_PATH;
if (!configured || !path.isAbsolute(configured)) {
throw new Error("Set OPENCLAW_QA_NODE_PATH to an absolute Node executable path");
}
await access(configured, constants.X_OK);
return configured;
}
return process.execPath;
}This prevents executing arbitrary 3. 🟠 Symlink-following arbitrary deletion risk in QA runtime staging (fs.rm on repo-controlled .artifacts path)
Description
Because the path is under a repo-controlled directory ( Key points:
Vulnerable code: const stagedRoot = path.join(
params.repoRoot,
".artifacts",
"qa-runtime",
path.basename(params.tempRoot),
);
await fs.rm(stagedRoot, { recursive: true, force: true });RecommendationHarden staging directory creation/deletion against symlink attacks:
Example mitigation (one approach): const artifactsDir = path.join(params.repoRoot, ".artifacts");
const qaRuntimeDir = path.join(artifactsDir, "qa-runtime");
for (const p of [artifactsDir, qaRuntimeDir]) {
if (existsSync(p)) {
const st = await fs.lstat(p);
if (st.isSymbolicLink()) throw new Error(`Refusing to use symlinked path: ${p}`);
if (!st.isDirectory()) throw new Error(`Expected directory: ${p}`);
}
}
await fs.mkdir(qaRuntimeDir, { recursive: true });
const stagedRoot = path.join(qaRuntimeDir, path.basename(params.tempRoot));
const realBase = await fs.realpath(qaRuntimeDir);
const realTargetParent = await fs.realpath(path.dirname(stagedRoot));
if (realTargetParent !== realBase) throw new Error("Unexpected stagedRoot parent");
await fs.rm(stagedRoot, { recursive: true, force: true });This prevents 4. 🟡 Dynamic import of private QA CLI from filesystem path derived from cwd/argv enables local code execution when env flag is set
Description
When
The code then imports that file URL and executes it. This turns the private QA flag into an untrusted search path mechanism that can execute arbitrary code from the filesystem. Vulnerable code: const packageRoot = resolvePackageRootSync({ argv1, cwd, moduleUrl });
...
const sourceModulePath = path.join(packageRoot, "dist/plugin-sdk/qa-lab.js");
...
return (await import(pathToFileURL(sourceModulePath).href)) as Record<string, unknown>;Notes:
RecommendationAvoid resolving the import target from attacker-influenceable locations (notably
Example hardening (remove const packageRoot = resolveOpenClawPackageRootSync({
moduleUrl: import.meta.url,
argv1: process.argv[1],
// no cwd fallback
});
if (!packageRoot) throw new Error("...");If you must allow Analyzed PR: #67428 at commit Last updated on: 2026-04-16T02:02:03Z |
There was a problem hiding this comment.
Pull request overview
Fixes private QA runtime/module resolution so local source checkouts and isolated QA gateway staging consistently load the intended plugin tree and runtime helpers.
Changes:
- Add a private, local-only plugin-sdk subpath list (gated by
OPENCLAW_ENABLE_PRIVATE_QA_CLI) and include it in plugin-sdk alias/subpath resolution and caching. - Improve QA discovery/staging for source checkouts and isolated gateway child runs (including Node exec-path resolution when parent runtime is bun).
- Update QA scenarios/flow runner to support
qaImport()in flow expressions and adjust scenarios accordingly.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/sdk-alias.ts | Adds gated private subpath listing + cache-keying for alias/subpath enumeration. |
| src/plugins/sdk-alias.test.ts | Adds tests for private QA subpaths and cache invalidation behavior. |
| src/plugin-sdk/root-alias.cjs | Extends root alias mapping to include private QA subpaths and adjusts alias precedence. |
| src/plugin-sdk/qa-runner-runtime.ts | Overrides manifest discovery env to prefer source extensions/ when in a repo checkout and private QA is enabled. |
| src/plugin-sdk/qa-runner-runtime.test.ts | Tests private QA discovery preferring the source bundled tree. |
| src/infra/run-node.test.ts | Adds coverage for skipping/requiring rebuilds for QA commands based on presence of the QA CLI facade artifact. |
| src/cli/program/register.subclis.test.ts | Updates mocking to the new QA CLI module specifier. |
| src/cli/program/private-qa-cli.ts | Switches private QA CLI module load to a package subpath specifier. |
| scripts/run-node.mjs | Points private QA dist entry check to dist/plugin-sdk/qa-lab.js. |
| scripts/lib/plugin-sdk-private-local-only-subpaths.json | Defines the private local-only plugin-sdk subpaths (qa-lab, qa-runtime). |
| qa/scenarios/session-memory-ranking.md | Writes stale memory into a directory instead of a single file path. |
| qa/scenarios/bundled-plugin-skill-runtime.md | Uses qaImport() for runtime imports inside flow expressions. |
| extensions/qa-lab/src/scenario-flow-runner.ts | Exposes qaImport() to flow expression evaluation context. |
| extensions/qa-lab/src/scenario-flow-runner.test.ts | Adds test validating qaImport() works in flow expressions. |
| extensions/qa-lab/src/qa-transport-registry.ts | Refactors transport registry and adds default concurrency plumbing. |
| extensions/qa-lab/src/qa-channel-transport.ts | Adds a default suite concurrency constant. |
| extensions/qa-lab/src/node-exec.ts | Adds Node executable resolution (notably for bun parent runtimes). |
| extensions/qa-lab/src/node-exec.test.ts | Tests Node exec-path resolution behavior and error messaging. |
| extensions/qa-lab/src/model-catalog.runtime.ts | Spawns model-catalog CLI using resolved Node exec path instead of process.execPath. |
| extensions/qa-lab/src/gateway-child.ts | Uses shared staging helpers and spawns gateway child with resolved Node exec path. |
| extensions/qa-lab/src/gateway-child.test.ts | Updates tests for new staging helpers and verifies import behavior from staged runtime roots. |
| extensions/qa-lab/src/bundled-plugin-staging.ts | New shared implementation for bundled plugin staging, owner resolution, and min host version detection. |
| extensions/qa-channel/src/runtime.ts | Updates runtime store creation to pass pluginId metadata. |
Greptile SummaryThis PR fixes private QA runtime source loading for repo checkouts. The main changes are: (1) extracting bundled plugin staging logic from Confidence Score: 5/5Safe to merge; changes are well-scoped to QA infrastructure, all key paths are covered by new tests, and no production API surfaces are broken. All findings are P2 or non-existent. The core logic (per-plugin dist/source fallback, Node exec resolution, alias cache invalidation) is correct and tests exercise the new source-checkout path including ESM import resolution through the staged openclaw package. No correctness or data-integrity issues found. No files require special attention. Reviews (1): Last reviewed commit: "QA: fix private runtime source loading" | Re-trigger Greptile |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc88642665
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd4235557e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 72ddcc98d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf296d62bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4cca44b64e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Fix the private QA wrapper and source-checkout runtime paths so qa-lab, qa-channel, and qa-matrix resolve their local-only SDK surfaces and staged bundled plugins reliably. This keeps private QA behavior local-only, restores source-runner discovery, and makes the isolated gateway runtime load the right plugin tree under Node.
6957704 to
b8bf2b6
Compare
|
Merged via squash.
Thanks @gumadeiras! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b8bf2b6be6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Merged via squash. Prepared head SHA: b8bf2b6 Co-authored-by: gumadeiras <[email protected]> Co-authored-by: gumadeiras <[email protected]> Reviewed-by: @gumadeiras
Summary
Testing
pnpm test src/infra/run-node.test.ts src/plugin-sdk/qa-runner-runtime.test.ts src/plugins/sdk-alias.test.ts src/cli/program/register.subclis.test.ts extensions/qa-lab/src/gateway-child.test.ts extensions/qa-lab/src/scenario-flow-runner.test.tspnpm buildpnpm openclaw qa matrix --provider-mode mock-openai --scenario matrix-thread-follow-up --output-dir .artifacts/qa-e2e/pr1-matrix-smokepnpm openclaw qa suite --transport qa-channel --provider-mode mock-openai --scenario bundled-plugin-skill-runtime --output-dir .artifacts/qa-e2e/pr1-qa-channel-smoke