Skip to content

Commit a5c40bf

Browse files
cliffhallclaude
andauthored
feat: port the skills harness (verify:skills, verify:skills:cli, skills:eval)
Port the MCP Inspector's skills infrastructure for #4858 Part 3: - scripts/verify-skills.mjs, verify-skills-cli.mjs, skill-eval.mjs and scripts/lib/{skill-manifest,claude-cli,npm-scripts,win-shell-args}.mjs, with all their *.test.mjs files - root npm scripts verify:skills, verify:skills:cli, skills:eval and test:scripts; yaml as a root devDependency - docs/skill-authoring.md, with paths and provenance adapted - a root-guards job in typescript.yml running verify:skills and verify:skills:cli on every push and pull request - .claude/skills/ (kept by a .gitkeep until the first skill lands) - un-ignore scripts/lib/, which the Python `lib/` pattern swallowed Adapted to this repo: checkWiring reads typescript.yml, and carries two temporary allowances (validate -> verify:format-coverage, removed by #4864; local:gate -> verify:skills:cli, removed by #4871) that excuse a link only while its pending script is absent and fail as stale once it exists. The "no skills found" failure is kept, with a temporary EMPTY_SKILLS_BOOTSTRAP allowance that fails as stale once a skill exists. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent da08675 commit a5c40bf

20 files changed

Lines changed: 6106 additions & 2 deletions

‎.claude/skills/.gitkeep‎

Whitespace-only changes.

‎.github/workflows/typescript.yml‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,33 @@ concurrency:
1111
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
1212

1313
jobs:
14+
# Root-level guards: checks over repo-wide tooling that no per-package matrix
15+
# leg covers. Created by #4863 for the skills gates; #4864 extends it with the
16+
# root format/lint, the coverage guards and `test:scripts`.
17+
# `scripts/verify-skills.mjs` asserts this job keeps running
18+
# `verify:skills:cli` unconditionally, so do not gate these steps with `if:`.
19+
root-guards:
20+
name: Root guards
21+
runs-on: ubuntu-latest
22+
steps:
23+
- uses: actions/checkout@v6
24+
25+
- uses: actions/setup-node@v7
26+
with:
27+
node-version: 22
28+
cache: npm
29+
30+
- name: Install dependencies
31+
run: npm ci
32+
33+
- name: Verify skills
34+
run: npm run verify:skills
35+
36+
# The authoritative `claude plugin validate`, at the pinned CLI version
37+
# (fetched with npx when not installed). Needs no authentication.
38+
- name: Validate skills with the pinned Claude Code CLI
39+
run: npm run verify:skills:cli
40+
1441
detect-packages:
1542
runs-on: ubuntu-latest
1643
outputs:
@@ -69,4 +96,3 @@ jobs:
6996
- name: Build package
7097
working-directory: src/${{ matrix.package }}
7198
run: npm run build
72-

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,3 +303,7 @@ cython_debug/
303303
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
304304
#.idea/
305305
.claude/settings.local.json
306+
307+
# The Python `lib/` pattern above would swallow the root tooling's shared
308+
# helpers; `scripts/lib/` is first-party source.
309+
!scripts/lib/

‎docs/skill-authoring.md‎

Lines changed: 594 additions & 0 deletions
Large diffs are not rendered by default.

‎package-lock.json‎

Lines changed: 19 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,21 @@
1616
"build": "npm run build --workspaces",
1717
"watch": "npm run watch --workspaces",
1818
"publish-all": "npm publish --workspaces --access public",
19-
"link-all": "npm link --workspaces"
19+
"link-all": "npm link --workspaces",
20+
"verify:skills": "node scripts/verify-skills.mjs",
21+
"verify:skills:cli": "node scripts/verify-skills-cli.mjs",
22+
"skills:eval": "node scripts/skill-eval.mjs",
23+
"test:scripts": "node --test \"scripts/**/*.test.mjs\""
2024
},
2125
"dependencies": {
2226
"@modelcontextprotocol/server-everything": "*",
2327
"@modelcontextprotocol/server-memory": "*",
2428
"@modelcontextprotocol/server-filesystem": "*",
2529
"@modelcontextprotocol/server-sequential-thinking": "*"
2630
},
31+
"devDependencies": {
32+
"yaml": "^2.9.0"
33+
},
2734
"overrides": {
2835
"qs": ">=6.15.2",
2936
"hono": ">=4.12.21"

‎scripts/lib/claude-cli.mjs‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
// Spawning the `claude` CLI from the root scripts (inspector#2163, Copilot).
2+
//
3+
// Ported from the MCP Inspector (#4863); `inspector#N` references are that
4+
// repo's issues and PRs, a bare `#N` is this repo's.
5+
//
6+
// On Windows an npm-installed `claude` is a `.cmd` shim, and Node has refused
7+
// shell-free `.cmd`/`.bat` spawns since the CVE-2024-27980 hardening — so a
8+
// plain `spawnSync("claude", …)` reports ENOENT there even though the CLI is
9+
// installed. Every consumer of that would then be wrong in a *quiet* way:
10+
// `verify:skills` would skip its authoritative hand-off, `verify:skills:cli`
11+
// would download the pinned package it already has (and fail offline), and
12+
// `skills:eval` would report "not on PATH".
13+
//
14+
// The shell that fixes it introduces the inspector#1939 problem — `cmd.exe` re-parses a
15+
// space-joined string, so any argument holding a metacharacter becomes syntax —
16+
// so arguments go through the same `winShellArgs` quoting the npm/npx call
17+
// sites use. This is deliberately the ONLY place that decides either question.
18+
//
19+
// The same two questions apply to every agent CLI the skills eval drives —
20+
// `copilot` is an npm-installed `.cmd` shim on Windows too (inspector#2397) — so the
21+
// decision is made once, for any command, and the `claude` helpers delegate.
22+
23+
import { spawnSync } from "node:child_process";
24+
import { winShellArgs } from "./win-shell-args.mjs";
25+
26+
/**
27+
* `spawnSync` arguments for a `claude` invocation, correct on every platform.
28+
*
29+
* @param {string[]} args
30+
* @param {object} [options] Passed through to `spawnSync`.
31+
* @param {string} [platform] Defaults to the current platform; injectable for tests.
32+
* @returns {{ command: string, args: string[], options: object }}
33+
*/
34+
export function claudeSpawnArgs(
35+
args,
36+
options = {},
37+
platform = process.platform,
38+
) {
39+
return cliSpawnArgs("claude", args, options, platform);
40+
}
41+
42+
/**
43+
* `spawn` arguments for any npm-installed agent CLI, correct on every platform.
44+
*
45+
* @param {string} command
46+
* @param {string[]} args
47+
* @param {object} [options] Passed through to the spawn call.
48+
* @param {string} [platform] Defaults to the current platform; injectable for tests.
49+
* @returns {{ command: string, args: string[], options: object }}
50+
*/
51+
export function cliSpawnArgs(
52+
command,
53+
args,
54+
options = {},
55+
platform = process.platform,
56+
) {
57+
return {
58+
command,
59+
args: winShellArgs(args, platform),
60+
options: { ...options, shell: platform === "win32" },
61+
};
62+
}
63+
64+
/**
65+
* Read the installed CLI's version, or null when there is no usable one.
66+
*
67+
* "Usable" deliberately includes parsing the output: a `claude` that answers
68+
* `--version` with something unrecognizable cannot be trusted to be the pinned
69+
* one either, and the callers all treat null as "use the pinned package".
70+
*
71+
* Generic in the parser's result: this helper only distinguishes "parsed" from
72+
* "did not", so the shape is whatever the caller's parser returns — today
73+
* `{ parts, prerelease }` from `parseClaudeVersion`. Hard-coding `number[]`
74+
* here made valid callers look type-invalid (Copilot).
75+
*
76+
* @template T
77+
* @param {(text: string) => T | null} parseVersion
78+
* @param {{ spawn?: typeof spawnSync, platform?: string }} [io]
79+
* @returns {T | null}
80+
*/
81+
export function probeClaudeVersion(parseVersion, io = {}) {
82+
return probeCliVersion("claude", parseVersion, io);
83+
}
84+
85+
/**
86+
* Read any agent CLI's version, or null when there is no usable one.
87+
*
88+
* @template T
89+
* @param {string} cli
90+
* @param {(text: string) => T | null} parseVersion
91+
* @param {{ spawn?: typeof spawnSync, platform?: string }} [io]
92+
* @returns {T | null}
93+
*/
94+
export function probeCliVersion(
95+
cli,
96+
parseVersion,
97+
{ spawn = spawnSync, platform } = {},
98+
) {
99+
const { command, args, options } = cliSpawnArgs(
100+
cli,
101+
["--version"],
102+
{ encoding: "utf8" },
103+
platform,
104+
);
105+
const res = spawn(command, args, options);
106+
if (res.error || res.status !== 0) return null;
107+
return parseVersion(res.stdout ?? "");
108+
}

‎scripts/lib/claude-cli.test.mjs‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
// Tests for the one place that decides how `claude` is spawned (inspector#2163, Copilot).
2+
//
3+
// The behavior under test only differs ON Windows, and these run on whatever
4+
// the contributor has — so the platform is injected rather than sniffed. A
5+
// win32-only bug that no CI runner here executes is precisely the kind that
6+
// ships.
7+
8+
import { test } from "node:test";
9+
import assert from "node:assert/strict";
10+
import {
11+
claudeSpawnArgs,
12+
cliSpawnArgs,
13+
probeClaudeVersion,
14+
probeCliVersion,
15+
} from "./claude-cli.mjs";
16+
17+
test("spawns without a shell off Windows", () => {
18+
const { command, args, options } = claudeSpawnArgs(
19+
["plugin", "validate", ".claude/skills"],
20+
{ cwd: "/repo" },
21+
"linux",
22+
);
23+
assert.equal(command, "claude");
24+
assert.deepEqual(args, ["plugin", "validate", ".claude/skills"]);
25+
assert.equal(options.shell, false);
26+
assert.equal(options.cwd, "/repo");
27+
});
28+
29+
test("uses a shell on Windows, where the CLI is a .cmd shim", () => {
30+
// Node has refused shell-free `.cmd` spawns since CVE-2024-27980, so without
31+
// this the CLI reads as absent even when it is installed.
32+
const { options } = claudeSpawnArgs(["--version"], {}, "win32");
33+
assert.equal(options.shell, true);
34+
});
35+
36+
test("quotes arguments that cmd.exe would otherwise re-parse", () => {
37+
const { args } = claudeSpawnArgs(["-p", "does it? (yes & no)"], {}, "win32");
38+
assert.deepEqual(args, ["-p", '"does it? (yes & no)"']);
39+
// The same arguments are untouched off Windows.
40+
assert.deepEqual(
41+
claudeSpawnArgs(["-p", "does it? (yes & no)"], {}, "linux").args,
42+
["-p", "does it? (yes & no)"],
43+
);
44+
});
45+
46+
test("probeClaudeVersion returns the parsed version on success", () => {
47+
const version = probeClaudeVersion(
48+
(t) => (t.trim() === "2.1.250" ? [2, 1, 250] : null),
49+
{
50+
spawn: () => ({ status: 0, stdout: "2.1.250" }),
51+
platform: "linux",
52+
},
53+
);
54+
assert.deepEqual(version, [2, 1, 250]);
55+
});
56+
57+
test("probeClaudeVersion returns null for every unusable outcome", () => {
58+
for (const res of [
59+
{ error: new Error("ENOENT") },
60+
{ status: 1, stdout: "" },
61+
{ status: 0, stdout: "something else" },
62+
]) {
63+
assert.equal(
64+
probeClaudeVersion(() => null, { spawn: () => res, platform: "linux" }),
65+
null,
66+
);
67+
}
68+
});
69+
70+
test("probeClaudeVersion asks for the shell on Windows", () => {
71+
let seen;
72+
probeClaudeVersion(() => [2, 1, 250], {
73+
spawn: (_c, _a, options) => {
74+
seen = options;
75+
return { status: 0, stdout: "2.1.250" };
76+
},
77+
platform: "win32",
78+
});
79+
assert.equal(seen.shell, true);
80+
assert.equal(seen.encoding, "utf8");
81+
});
82+
83+
test("the generic helpers name the command they were given (inspector#2397)", () => {
84+
// `copilot` is an npm `.cmd` shim on Windows as well, so it takes the same
85+
// shell-and-quoting decision rather than a second copy of it.
86+
const { command, options, args } = cliSpawnArgs(
87+
"copilot",
88+
["--prompt", "a & b"],
89+
{},
90+
"win32",
91+
);
92+
assert.equal(command, "copilot");
93+
assert.equal(options.shell, true);
94+
assert.deepEqual(args, ["--prompt", '"a & b"']);
95+
96+
let spawned;
97+
const version = probeCliVersion("copilot", (t) => t.trim(), {
98+
spawn: (c) => {
99+
spawned = c;
100+
return { status: 0, stdout: "GitHub Copilot CLI 1.0.85.\n" };
101+
},
102+
platform: "linux",
103+
});
104+
assert.equal(spawned, "copilot");
105+
assert.equal(version, "GitHub Copilot CLI 1.0.85.");
106+
});

0 commit comments

Comments
 (0)