Skip to content

Commit 593a75e

Browse files
thymikeeclaude
andcommitted
fix(cli): apply review notes on the failure summary line
- Read the --debug log tail from the end of the file, bounded in bytes, instead of reading the whole daemon log into memory. - Repeat the full message in the body only when the summary cut it. - commandSynopsis rethrows anything other than a retired-name AppError. - Assert the full summary line in the writer test. - Move the eval measurements to docs/evidence, keep one line of rationale in ADR 0010, and add the rule to its glance list. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 96e1cd0 commit 593a75e

8 files changed

Lines changed: 64 additions & 18 deletions

File tree

‎docs/adr/0010-error-system.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
- If the per-code default hint would mislead, supply a hint. If the default suffices, omit the hint.
1717
- When you wrap an error, preserve an existing hint and the cause chain.
1818
- Every consumer surface must carry at least code, message, and hint. Rehydration helpers copy all fields.
19+
- A failed CLI command prints one summary line (code, message, dispatch fact, Next step) first and, when anything follows it, last. The diagnostic id is not printed.
1920
- The router's thrown-error finalizer attempts a diagnostics flush. `diagnosticId` requires an active request diagnostics scope, and `logPath` appears only when that flush produces a file. Returned or pre-scope failures may omit both fields. Messages must not embed raw stderr dumps or secrets.
2021

2122
## Context
@@ -103,10 +104,8 @@ Nothing documented the conventions, so each new site decided them again.
103104
itself depends on the code only.
104105
The body between the copies holds the cause, warnings, candidates, the `Log:` path, and the
105106
`--debug` details and log tail. The diagnostic id is not printed; it stays in `--json`.
106-
Reason: agents merge stderr into stdout and keep `head -1` or, most often, `tail -1`. A model
107-
eval showed that a footer-last layout hides the code under `tail -1`, that one summary without the
108-
repeat recovers poorly under `tail -1`, and that models follow `Next` even when it is wrong unless
109-
the dispatch fact on the same line contradicts it.
107+
Reason: agents merge stderr into stdout and keep `head -1` or, most often, `tail -1`; a footer-last
108+
layout hides the code under `tail -1`. Measurements: `docs/evidence/cli-failure-summary-eval-2026-11-07.md`.
110109
Consequence: a default hint is printed as an instruction, so `defaultHintForCode` never asks for a
111110
`--debug` rerun when a log already exists, and any retry a hint implies follows the observe-first
112111
step whenever the request may have dispatched.
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# CLI failure summary line evaluation, 2026-11-07
2+
3+
Small models recovered correctly from every failure when stderr opened and closed with one summary line that carried the error code, the dispatch fact and a runnable Next step. With the layout before the change, 58% recovered. Without the closing copy of the summary, recovery under `tail -1` fell to 11%. ADR 0010 decision 7 records the resulting rule.
4+
5+
These are synthetic renders of real failure scenarios, not live CLI runs. The author wrote the Next text and the answer key, so the 100% result partly measures whether a model follows a correct Next step. The wrong-Next condition below measures that directly.
6+
7+
## How agents trim output
8+
9+
Of 529 agent-device calls in recent agent transcripts that piped output through `head` or `tail`, 520 merged stderr into stdout with `2>&1`. `tail -1` was the most common cut (257), then `head -20..60` (about 100), `tail -2` (44), `tail -3` (20) and `head -1` (18). Of the `tail -1` calls, 184 were `agent-device daemon` calls in development sessions; about 90 device commands used `tail -1..3`. Many entries are repeated strings, so treat these as rough frequencies.
10+
11+
## Method
12+
13+
Twelve failure scenarios used codes, messages and hints taken from the codebase: a timed-out tap, a partial `type`, a partial `scroll`, ambiguous matches with 2 and 6 candidates, a selector without a match, no session, invalid arguments, a missing app, a busy device, a missing tool and a failed read-only snapshot. Each output was cut four ways: full, `tail -1`, `tail -3` and `head -1`. Each unique prompt went to Claude Haiku 5.5, GPT-6 Luna, GPT-6.1 Terra and Apex at low effort, in a fresh headless session from an empty directory. The grader classified the next command as recovered, investigated (read the log or rerun with `--debug`), unsafe (re-sent a possibly dispatched action), stuck or other. Each cell ran once.
14+
15+
| Layout | Recovered | Unsafe | Stuck | Retry judgment right | Error code seen |
16+
| --- | ---: | ---: | ---: | ---: | ---: |
17+
| A: before the change, footer last | 58% | 7% | 14% | 81% | 57% |
18+
| B: footer moved behind `--debug` | 71% | 8% | 12% | 83% | 64% |
19+
| C: summary first and last, hints unchanged | 69% | 6% | 13% | 98% | 95% |
20+
| D: C with a runnable Next step | 100% | 0% | 0% | 97% | 96% |
21+
22+
Under `tail -1`, the error code was visible in 0% of A and B answers and in 100% of C and D answers. C recovered little better than B because the old generic hint ("Retry with --debug") became an instruction once it was labelled Next; under C, all three models that saw only the summary line reran a selector miss with `--debug`.
23+
24+
## Isolating the repeat and blind obedience
25+
26+
E printed D's content with the summary once, at the top (multi-line scenarios only, 28 answers per cut):
27+
28+
| Cut | D, summary repeated | E, summary once |
29+
| --- | ---: | ---: |
30+
| full | 100% | 100% |
31+
| `tail -3` | 100% | 75% |
32+
| `tail -1` | 100% | 11% |
33+
34+
F used D's layout with a deliberately wrong Next step. Models followed it in 78 of 192 answers. They never followed a re-tap that contradicted "May have reached the device" on the same line (0/16), and they rarely followed one that contradicted an argument or session error (0/16 and 1/16). They always followed a wrong step that nothing on the line contradicted, including `agent-device close` on a busy device (16/16) and `agent-device home` after a failed read (16/16).

‎packages/command-output/src/output/error.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ function formatSummary(
7676
...(dispatched ? [DISPATCH_SENTENCE[dispatched]] : []),
7777
...(next ? [`Next: ${next}`] : []),
7878
].join(' ');
79-
return { summary, truncated: shown !== normalized.message };
79+
return { summary, truncated: shown !== message };
8080
}
8181

8282
/**

‎packages/kernel/src/errors.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -532,7 +532,7 @@ function stringDetail(
532532
return typeof value === 'string' ? value : undefined;
533533
}
534534

535-
/** The producer-proved {@link DispatchDisclosure} in a details bag, or `undefined` when unclassified. */
535+
/** The recorded {@link DispatchDisclosure} in a details bag, or `undefined` when no layer recorded one. */
536536
export function readDispatchDisclosure(
537537
details: Record<string, unknown> | undefined,
538538
): DispatchDisclosure | undefined {

‎src/cli/failure-output.ts‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -82,14 +82,26 @@ function failureLogTail(options: {
8282
function logFileTail(label: string, logPath: string): string | undefined {
8383
try {
8484
if (!fs.existsSync(logPath)) return undefined;
85-
const content = fs.readFileSync(logPath, 'utf8');
86-
const lines = content.split('\n');
87-
let tail = lines.slice(Math.max(0, lines.length - 200)).join('\n');
88-
if (tail.length > DAEMON_LOG_TAIL_MAX_BYTES) {
89-
tail = tail.slice(tail.length - DAEMON_LOG_TAIL_MAX_BYTES);
90-
}
85+
const tail = readTrailingLines(logPath, DAEMON_LOG_TAIL_MAX_BYTES, 200);
9186
return tail.trim().length > 0 ? `\n[${label}]\n${tail.trimEnd()}` : undefined;
9287
} catch {
9388
return undefined;
9489
}
9590
}
91+
92+
/** The last `maxLines` lines within the file's last `maxBytes` bytes; the daemon log grows unbounded. */
93+
function readTrailingLines(filePath: string, maxBytes: number, maxLines: number): string {
94+
const fd = fs.openSync(filePath, 'r');
95+
try {
96+
const size = fs.fstatSync(fd).size;
97+
const length = Math.min(size, maxBytes);
98+
const buffer = Buffer.alloc(length);
99+
fs.readSync(fd, buffer, 0, length, size - length);
100+
const lines = buffer.toString('utf8').split('\n');
101+
// A read that starts mid-file starts mid-line (or mid-character); drop that partial line.
102+
if (length < size) lines.shift();
103+
return lines.slice(-maxLines).join('\n');
104+
} finally {
105+
fs.closeSync(fd);
106+
}
107+
}

‎src/cli/parser/args.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -466,8 +466,9 @@ export async function commandSynopsis(command: string): Promise<string | null> {
466466
const { buildCommandSynopsis } = await import('../../commands/schema/cli-help.ts');
467467
try {
468468
return buildCommandSynopsis(normalizeCommandAlias(command));
469-
} catch {
470-
return null;
469+
} catch (error) {
470+
if (error instanceof AppError) return null;
471+
throw error;
471472
}
472473
}
473474

‎src/commands/output/human-error.test.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ test('printHumanError closes a multi-line failure with the summary, after any tr
2727
});
2828

2929
const lines = stderr.trimEnd().split('\n');
30-
assert.match(
31-
lines[0] ?? '',
32-
/^Error \(COMMAND_FAILED\): boom\. May have reached the device\. Next: /,
30+
assert.equal(
31+
lines[0],
32+
'Error (COMMAND_FAILED): boom. May have reached the device. Next: Run snapshot to check the screen before you retry. Read the diagnostics log for the failing step.',
3333
);
3434
assert.equal(lines.at(-1), lines[0]);
3535
assert.ok(lines.indexOf('last daemon line') < lines.length - 1);

‎website/docs/docs/debugging-profiling.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@ Log: /Users/me/.agent-device/sessions/<session>/requests/<request-id>.ndjson
140140
Error (COMMAND_FAILED): tap @e12: runner did not answer within 10s. May have reached the device. Next: Run snapshot to check the screen before you retry. Read the diagnostics log for the failing step.
141141
```
142142
143-
The summary says whether the action reached the device. `May have reached the device` means the action can already have run, so check the screen before you retry it. `Not dispatched` means a retry is safe. A summary with neither does not prove that the action did not run. `Next` is the step to take. The `Log` line names the failing request's diagnostics file.
143+
When agent-device knows whether the action reached the device, the summary says so. `May have reached the device` means the action can already have run, so check the screen before you retry it. `Not dispatched` means a retry is safe. A summary with neither does not prove that the action did not run. `Next` is the step to take. The `Log` line names the failing request's diagnostics file.
144144
145145
Use `--json` to inspect `error.details.stderr` when it is present. Diagnostic stderr is secret-redacted and retains up to 8,192 characters; longer output keeps its beginning and end with a truncation marker between them. Other diagnostic strings remain limited to 400 characters. Compact retry summaries may retain less output.
146146

0 commit comments

Comments
 (0)