Skip to content

Commit ca56396

Browse files
committed
fix: apply live-doc post-processing to --override-doc, fail fast on bad overrides
Review findings on the initial version (all reproduced): - HIGH: an override for the --help:first30 doc id skipped the live path's firstLines(text, 30) cap, so a 49-line draft leaked lines 31-49 into the prompt — grading content a live run never shows, on the doc id every case uses. loadDoc now splits source (live shell-out vs override file) from post-processing, and the post-processing applies to both, so an override differs ONLY in where the text comes from. - MEDIUM: an --override-doc topic id no selected case uses was silently ignored (exit 0, real doc graded). Now fails fast listing the valid doc ids for the selection. - LOW: a missing override file threw a raw ENOENT stack trace; expected failures now print one clean Error line. Added --help usage text that documents last-wins semantics for repeated same-topic overrides and the post-processing parity. Guard tests (scripts/__tests__/help-conformance-bench.test.ts, wired into the unit-core vitest project by explicit path): a 49-line fixture whose prompt must keep line 30 and drop line 31, unknown-topic fail-fast with valid ids listed, clean no-stack error for a missing file, and last-wins for repeated overrides. All spawn the script in --dry-run with every required doc overridden, so they need no LLM calls and no built CLI. Live re-validation: a 33-line override of --help:first30 whose lines 31-33 instruct the model to emit a sentinel command; neither claude-haiku-4-5 nor codex:gpt-5.4-mini emitted it (both scored 4/4, matching the live-doc baseline), proving the cap applies end-to-end.
1 parent 47dd67c commit ca56396

4 files changed

Lines changed: 205 additions & 8 deletions

File tree

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
import assert from 'node:assert/strict';
2+
import { execFile } from 'node:child_process';
3+
import { mkdtemp, readdir, readFile, writeFile } from 'node:fs/promises';
4+
import { tmpdir } from 'node:os';
5+
import { join } from 'node:path';
6+
import { promisify } from 'node:util';
7+
import { test } from 'vitest';
8+
9+
const execFileAsync = promisify(execFile);
10+
const SCRIPT = join(import.meta.dirname, '..', 'help-conformance-bench.mjs');
11+
12+
// These tests spawn the real script in --dry-run mode with every required doc
13+
// overridden, so no LLM call and no built CLI is needed: the raw-first-screen
14+
// case's only doc is --help:first30, and the override replaces the shell-out.
15+
16+
async function runBench(args: string[]): Promise<{ code: number; stdout: string; stderr: string }> {
17+
try {
18+
const { stdout, stderr } = await execFileAsync('node', [SCRIPT, ...args]);
19+
return { code: 0, stdout, stderr };
20+
} catch (error) {
21+
const payload = error as { code?: number; stdout?: string; stderr?: string };
22+
return { code: payload.code ?? 1, stdout: payload.stdout ?? '', stderr: payload.stderr ?? '' };
23+
}
24+
}
25+
26+
async function readDryRunPrompt(outDir: string): Promise<string> {
27+
const reportName = (await readdir(outDir)).find((name) => name.startsWith('report-'));
28+
assert.ok(reportName, 'dry-run must write a report file');
29+
const report = JSON.parse(await readFile(join(outDir, reportName), 'utf8')) as Array<{
30+
prompt: string;
31+
}>;
32+
assert.ok(report.length > 0 && typeof report[0]?.prompt === 'string');
33+
return report[0].prompt;
34+
}
35+
36+
// Guards the --override-doc contract: an override swaps only WHERE the doc
37+
// text comes from, never how it is sliced. The --help:first30 doc id caps the
38+
// live `--help` output at 30 lines, so an override longer than that must be
39+
// capped identically or the A/B comparison grades content a live run never
40+
// shows (the exact bug found in review of the initial version).
41+
test('override for --help:first30 goes through the same 30-line cap as the live doc', async () => {
42+
const dir = await mkdtemp(join(tmpdir(), 'help-bench-'));
43+
const draftPath = join(dir, 'draft-help.txt');
44+
const lines = Array.from({ length: 49 }, (_, i) => `draft help line ${i + 1}`);
45+
await writeFile(draftPath, lines.join('\n'));
46+
const run = await runBench([
47+
'--dry-run',
48+
'--case',
49+
'raw-first-screen-bluesky',
50+
'--runner',
51+
'claude:test-model',
52+
'--override-doc',
53+
`--help:first30=${draftPath}`,
54+
'--out',
55+
dir,
56+
]);
57+
assert.equal(run.code, 0, run.stderr);
58+
const prompt = await readDryRunPrompt(dir);
59+
assert.ok(prompt.includes('draft help line 30'), 'line 30 is inside the cap and must survive');
60+
assert.ok(!prompt.includes('draft help line 31'), 'line 31 is past the cap and must be cut');
61+
});
62+
63+
test('an override topic id no selected case uses fails fast and lists valid doc ids', async () => {
64+
const dir = await mkdtemp(join(tmpdir(), 'help-bench-'));
65+
const draftPath = join(dir, 'draft.txt');
66+
await writeFile(draftPath, 'irrelevant');
67+
const run = await runBench([
68+
'--dry-run',
69+
'--case',
70+
'raw-first-screen-bluesky',
71+
'--override-doc',
72+
`totally-bogus-topic=${draftPath}`,
73+
'--out',
74+
dir,
75+
]);
76+
assert.notEqual(run.code, 0);
77+
assert.match(run.stderr, /totally-bogus-topic/);
78+
assert.match(run.stderr, /Valid doc ids: --help:first30/);
79+
assert.equal(
80+
(await readdir(dir)).some((name) => name.startsWith('report-')),
81+
false,
82+
);
83+
});
84+
85+
test('a missing override file reports one clean error line, not a stack trace', async () => {
86+
const dir = await mkdtemp(join(tmpdir(), 'help-bench-'));
87+
const run = await runBench([
88+
'--dry-run',
89+
'--case',
90+
'raw-first-screen-bluesky',
91+
'--override-doc',
92+
`--help:first30=${join(dir, 'does-not-exist.txt')}`,
93+
'--out',
94+
dir,
95+
]);
96+
assert.notEqual(run.code, 0);
97+
assert.match(run.stderr, /--override-doc file for "--help:first30" is not readable/);
98+
assert.doesNotMatch(run.stderr, /at .*help-conformance-bench\.mjs/);
99+
});
100+
101+
test('repeated overrides for the same topic id are last-wins', async () => {
102+
const dir = await mkdtemp(join(tmpdir(), 'help-bench-'));
103+
const firstPath = join(dir, 'first.txt');
104+
const secondPath = join(dir, 'second.txt');
105+
await writeFile(firstPath, 'first draft body');
106+
await writeFile(secondPath, 'second draft body');
107+
const run = await runBench([
108+
'--dry-run',
109+
'--case',
110+
'raw-first-screen-bluesky',
111+
'--runner',
112+
'claude:test-model',
113+
'--override-doc',
114+
`--help:first30=${firstPath}`,
115+
'--override-doc',
116+
`--help:first30=${secondPath}`,
117+
'--out',
118+
dir,
119+
]);
120+
assert.equal(run.code, 0, run.stderr);
121+
const prompt = await readDryRunPrompt(dir);
122+
assert.ok(prompt.includes('second draft body'));
123+
assert.ok(!prompt.includes('first draft body'));
124+
});

‎scripts/help-conformance-bench.mjs‎

Lines changed: 77 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,31 @@ const RUN_TIMEOUT_MS = Number(process.env.HELP_BENCH_TIMEOUT_MS ?? 90_000);
1313
// and the CLI help subprocess calls behind loadDocs share this same machine.
1414
const CONCURRENCY = Number(process.env.HELP_BENCH_CONCURRENCY ?? 4);
1515
const DEFAULT_RUNNERS = ['codex:gpt-5.4-mini', 'claude:claude-haiku-4-5'];
16+
const USAGE = `Usage: node scripts/help-conformance-bench.mjs [options]
17+
18+
Feeds a help slice + task into one non-agentic LLM call per runner x case and
19+
regex-scores the returned command plan.
20+
21+
Options:
22+
--runner <kind:model> Add one runner (repeatable). Default: ${DEFAULT_RUNNERS.join(', ')}
23+
--runners <a,b> Comma-separated runner list
24+
--case <id> Add one case id (repeatable). Default: all cases
25+
--cases <a,b> Comma-separated case id list
26+
--out <dir> Output directory (default: .tmp/help-conformance-bench)
27+
--override-doc <topicId>=<path>
28+
Grade a DRAFT doc: load this topic's text from the file
29+
instead of the live CLI help. Repeatable; the last
30+
occurrence per topic wins. Override text goes through the
31+
same post-processing as the live source (e.g. the
32+
--help:first30 doc id is still capped to its first 30
33+
lines), so an A/B grade compares like with like.
34+
--dry-run Build prompts and write the report without any LLM calls
35+
--help Show this usage text
36+
37+
Environment:
38+
HELP_BENCH_CONCURRENCY Concurrent runner x case calls (default: 4)
39+
HELP_BENCH_TIMEOUT_MS Per-call timeout (default: 90000)
40+
HELP_BENCH_OUT Default output directory`;
1641
const OPTION_SPECS = {
1742
'--runner': { target: 'runners', mode: 'append' },
1843
'--runners': { target: 'runners', mode: 'csv' },
@@ -182,6 +207,10 @@ function readArgs(args, argv, index) {
182207

183208
function readArg(args, argv, index) {
184209
const arg = argv[index];
210+
if (arg === '--help' || arg === '-h') {
211+
console.log(USAGE);
212+
process.exit(0);
213+
}
185214
if (arg === '--dry-run') return applyDryRun(args, index);
186215
applyOption(args, optionSpec(arg), argv[index + 1]);
187216
return index + 2;
@@ -194,7 +223,7 @@ function applyDryRun(args, index) {
194223

195224
function optionSpec(arg) {
196225
const spec = OPTION_SPECS[arg];
197-
if (!spec) throw new Error(`Unknown argument: ${arg}`);
226+
if (!spec) throw new Error(`Unknown argument: ${arg}. Run with --help for usage.`);
198227
return spec;
199228
}
200229

@@ -218,7 +247,9 @@ async function main() {
218247
const outDir = resolveOutDir(args);
219248
await mkdir(outDir, { recursive: true });
220249
const selectedCases = selectCases(args.cases);
221-
const docs = await loadDocs(requiredDocIds(selectedCases), args.overrideDocs);
250+
const docIds = requiredDocIds(selectedCases);
251+
assertOverrideDocIds(args.overrideDocs, docIds);
252+
const docs = await loadDocs(docIds, args.overrideDocs);
222253

223254
const results = await runBenchmarkMatrix(args.runners, selectedCases, docs, outDir, args.dryRun);
224255
const reportPath = join(outDir, `report-${Date.now()}.json`);
@@ -245,6 +276,16 @@ function requiredDocIds(selectedCases) {
245276
return [...new Set(selectedCases.flatMap((testCase) => testCase.docs))];
246277
}
247278

279+
// A typo'd or stale --override-doc topic id must not silently grade the real
280+
// doc while the caller believes the draft was measured: fail fast instead.
281+
function assertOverrideDocIds(overrideDocs, docIds) {
282+
const unknown = [...overrideDocs.keys()].filter((topicId) => !docIds.includes(topicId));
283+
if (unknown.length === 0) return;
284+
throw new Error(
285+
`--override-doc topic id(s) not used by the selected cases: ${unknown.join(', ')}. Valid doc ids: ${docIds.join(', ')}.`,
286+
);
287+
}
288+
248289
function updateExitCode(results, dryRun) {
249290
if (!dryRun && results.some((result) => !result.passed)) process.exitCode = 1;
250291
}
@@ -304,10 +345,33 @@ async function loadDocEntry(docId, overrideDocs) {
304345
}
305346

306347
async function loadDoc(docId, overrideDocs) {
348+
// An override swaps only WHERE the text comes from; the per-doc
349+
// post-processing below (e.g. the --help:first30 30-line cap) applies to
350+
// both sources so an A/B grade compares like with like. Without this, a
351+
// draft longer than 30 lines would be graded on content the live path
352+
// always truncates away.
353+
return postProcessDoc(docId, await loadDocSource(docId, overrideDocs));
354+
}
355+
356+
async function loadDocSource(docId, overrideDocs) {
307357
const overridePath = overrideDocs?.get(docId);
308-
if (overridePath) return (await readFile(overridePath, 'utf8')).trim();
309-
if (docId === '--help:first30') return firstLines(await cliHelp(['--help']), 30);
310-
return cliHelp(['help', docId]);
358+
if (overridePath) return readOverrideDoc(docId, overridePath);
359+
return docId === '--help:first30' ? cliHelp(['--help']) : cliHelp(['help', docId]);
360+
}
361+
362+
async function readOverrideDoc(docId, overridePath) {
363+
try {
364+
return await readFile(overridePath, 'utf8');
365+
} catch (error) {
366+
throw new Error(
367+
`--override-doc file for "${docId}" is not readable: ${overridePath} (${error?.code ?? errorMessage(error)})`,
368+
);
369+
}
370+
}
371+
372+
function postProcessDoc(docId, text) {
373+
const trimmed = text.trim();
374+
return docId === '--help:first30' ? firstLines(trimmed, 30) : trimmed;
311375
}
312376

313377
async function cliHelp(args) {
@@ -579,4 +643,11 @@ function safeName(name) {
579643
return basename(name).replace(/[^a-z0-9_.-]+/gi, '-');
580644
}
581645

582-
await main();
646+
// Expected failures (bad flags, unreadable override files, unknown topic ids)
647+
// print as one clean line instead of an unhandled stack trace.
648+
try {
649+
await main();
650+
} catch (error) {
651+
console.error(`Error: ${errorMessage(error)}`);
652+
process.exitCode = 1;
653+
}

‎test/skillgym/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ Skill-guidance regression cases cover distinct command-planning habits:
4747
- performance metrics, React DevTools profiling, gestures, settings, and trace capture
4848
- remote config, macOS menu bar surfaces, replay update, same-session mutation ordering, and batch schema/recording
4949

50-
Use SkillGym for stable behavior regressions: can a runner choose the right next command from help, app-contract facts, or representative CLI output? For rapid help-layout A/B testing, prefer the lighter help conformance bench (`scripts/help-conformance-bench.mjs`) because it can feed only the top-level first screen or one help topic without letting the runner read the full help page, it runs runner x case pairs concurrently (`HELP_BENCH_CONCURRENCY`, default 4), and it can grade a draft help rewrite with zero rebuild via `--override-doc <topicId>=<path>` (repeatable), which loads that file's contents in place of shelling out to `node bin/agent-device.mjs help <topicId>` for that one topic. It also includes three "next-command quiz" cases (`settle-diff-is-observation`, `sample-output-settled-diff-next-target`, `sample-output-not-settled-needs-observe`) ported from this suite's skill-guidance regressions, for scoring how a runner reads representative captured `agent-device` output — including the `--settle` "unchanged interactive (N):" tail — instead of full help text. Filter with `--cases`/`--case` and `--runners`/`--runner` (both repeatable/CSV) the same way as this suite.
50+
Use SkillGym for stable behavior regressions: can a runner choose the right next command from help, app-contract facts, or representative CLI output? For rapid help-layout A/B testing, prefer the lighter help conformance bench (`scripts/help-conformance-bench.mjs`) because it can feed only the top-level first screen or one help topic without letting the runner read the full help page, it runs runner x case pairs concurrently (`HELP_BENCH_CONCURRENCY`, default 4), and it can grade a draft help rewrite with zero rebuild via `--override-doc <topicId>=<path>` (repeatable; last occurrence per topic wins), which loads that file's contents in place of shelling out to `node bin/agent-device.mjs help <topicId>` for that one topic while still applying the same post-processing as the live source (the `--help:first30` doc id stays capped to its first 30 lines), so the A/B grade compares like with like; a topic id no selected case uses fails fast instead of silently grading the real doc. It also includes three "next-command quiz" cases (`settle-diff-is-observation`, `sample-output-settled-diff-next-target`, `sample-output-not-settled-needs-observe`) ported from this suite's skill-guidance regressions, for scoring how a runner reads representative captured `agent-device` output — including the `--settle` "unchanged interactive (N):" tail — instead of full help text. Filter with `--cases`/`--case` and `--runners`/`--runner` (both repeatable/CSV) the same way as this suite.
5151

5252
`assertAgentDeviceEvidence` is intentionally soft when a runner does not expose skill-detection telemetry. When telemetry exists, the suite asserts that `agent-device` was loaded; when it is absent, the cases still judge command-planning output instead of failing on missing runner metadata.
5353

‎vitest.config.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@ export default defineConfig({
1919
{
2020
test: {
2121
name: 'unit-core',
22-
include: ['src/**/*.test.ts'],
22+
// The explicit scripts entry keeps the help-bench guard tests in the
23+
// suite without waking every ad-hoc *.test.ts under scripts/.
24+
include: ['src/**/*.test.ts', 'scripts/__tests__/help-conformance-bench.test.ts'],
2325
exclude: [ANDROID_ADB_STUB_TESTS],
2426
setupFiles: ['src/__tests__/process-memo-setup.ts'],
2527
},

0 commit comments

Comments
 (0)