Skip to content

Commit 6a5076d

Browse files
committed
fix(replay): surface warnings on failed runs and keep them honest
A failing `replay` still printed no trace of the run's accumulated warnings: the human error renderer read message, hint, and divergence but never the error-level warnings channel, so the pass/fail asymmetry survived on that surface. It now prints each warning after the error. Round-two corrections: warnings on the divergence response are scrubbed against replay vars like message and hint already are; a failed attempt copies warnings only when non-empty, matching the passing branch; the optional-skip hint append is removed because no producer of a maestro test failure emits a hint — capture-error lineage stays with #2450 — along with the CHANGELOG claim it rested on; the failed-result copy gains its own module test, and JUnit covers a failed test's warning lines.
1 parent 62e2dcc commit 6a5076d

10 files changed

Lines changed: 152 additions & 39 deletions

File tree

‎packages/maestro/src/internal/__tests__/engine.test.ts‎

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -366,30 +366,6 @@ describe('executeMaestroProgram', () => {
366366
);
367367
});
368368

369-
test('optional-skip warnings append the failure hint', async () => {
370-
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
371-
request.invalidateObservation();
372-
throw maestroTestFailure('Maestro target did not resolve to a visible element.', {
373-
hint: 'Regular presentation requires valid viewport evidence.',
374-
});
375-
});
376-
const port = makePort({
377-
observe: vi.fn(async ({ generation }) => ({ generation, matched: false })),
378-
execute,
379-
});
380-
const program = parseMaestroProgram(
381-
['---', '- tapOn:', ' text: Missing target', ' optional: true'].join('\n'),
382-
);
383-
384-
const result = await executeMaestroProgram(program, port);
385-
386-
expect(result.warnings).toEqual([
387-
expect.stringMatching(
388-
/Optional Maestro tapOn skipped at line 2: .*\(Regular presentation requires valid viewport evidence\.\)/,
389-
),
390-
]);
391-
});
392-
393369
test('observer failure cannot mask nested leaf failure provenance', async () => {
394370
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
395371
request.invalidateObservation();

‎packages/maestro/src/internal/replay-plan-step-execution.ts‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -99,13 +99,8 @@ async function executeOptionalCommand(
9999

100100
function formatOptionalWarning(command: MaestroRuntimeCommand, error: unknown): string {
101101
const source = `${command.source.path ? `${command.source.path}:` : ''}line ${command.source.line}`;
102-
const detail =
103-
error instanceof AppError
104-
? typeof error.details?.hint === 'string'
105-
? `${error.message} (${error.details.hint})`
106-
: error.message
107-
: String(error);
108-
return `Optional Maestro ${command.kind} skipped at ${source}: ${detail}`;
102+
const message = error instanceof AppError ? error.message : String(error);
103+
return `Optional Maestro ${command.kind} skipped at ${source}: ${message}`;
109104
}
110105

111106
function isOptionalCommand(command: MaestroRuntimeCommand): boolean {
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
import fs from 'node:fs';
2+
import os from 'node:os';
3+
import path from 'node:path';
4+
import { expect, test } from 'vitest';
5+
import { runReplayTestCase } from '../session-test-attempt.ts';
6+
import type { ReplayTestRunEntry } from '../session-test-discovery.ts';
7+
import type { ReplayTestAttemptOutcome } from '../session-test-types.ts';
8+
import type { ReplaySuiteTestFailed } from '@agent-device/contracts/replay';
9+
10+
const FAILED_WITH_WARNINGS: ReplayTestAttemptOutcome = {
11+
status: 'failed',
12+
error: {
13+
code: 'REPLAY_DIVERGENCE',
14+
message: 'Replay failed at step 2 (tapOn "Save"): target did not resolve',
15+
},
16+
artifactPaths: [],
17+
warnings: ['Optional Maestro assertVisible skipped at line 3: sheet owns focus'],
18+
infrastructure: false,
19+
};
20+
21+
const FAILED_WITHOUT_WARNINGS: ReplayTestAttemptOutcome = {
22+
status: 'failed',
23+
error: { code: 'COMMAND_FAILED', message: 'tap failed' },
24+
artifactPaths: [],
25+
infrastructure: false,
26+
};
27+
28+
function makeEntry(): ReplayTestRunEntry {
29+
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-test-attempt-'));
30+
const filePath = path.join(root, '01-flow.ad');
31+
fs.writeFileSync(filePath, 'context platform=ios\nopen "Demo"\n');
32+
return {
33+
kind: 'run',
34+
path: filePath,
35+
title: 'flow',
36+
manifest: { device: { platform: { kind: 'declared', value: 'ios' } } },
37+
};
38+
}
39+
40+
async function runFailedCase(outcome: ReplayTestAttemptOutcome): Promise<ReplaySuiteTestFailed> {
41+
const report = await runReplayTestCase({
42+
entry: makeEntry(),
43+
sessionName: 'default',
44+
suiteInvocationId: 'suite-attempt',
45+
caseIndex: 0,
46+
retries: 0,
47+
suiteArtifactsDir: fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-test-attempt-suite-')),
48+
suiteIndex: 1,
49+
suiteTotal: 1,
50+
runReplay: async () => outcome,
51+
cleanupSession: async () => {},
52+
emitProgress: () => {},
53+
isCanceled: () => false,
54+
emitDiagnostic: () => {},
55+
bindAttemptCancellation: () => ({ cancel: () => {}, release: () => {} }),
56+
});
57+
if (report.result.status !== 'failed') throw new Error('expected a failed test result');
58+
return report.result;
59+
}
60+
61+
test('a failed test result carries the warnings accumulated before the failure (#2560)', async () => {
62+
const result = await runFailedCase(FAILED_WITH_WARNINGS);
63+
expect(result.warnings).toEqual([
64+
'Optional Maestro assertVisible skipped at line 3: sheet owns focus',
65+
]);
66+
});
67+
68+
test('a failed test result omits warnings when the attempt had none', async () => {
69+
const result = await runFailedCase(FAILED_WITHOUT_WARNINGS);
70+
expect('warnings' in result).toBe(false);
71+
});

‎packages/replay-test/src/internal/session-test-attempt.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -388,8 +388,8 @@ function buildReplayTestFailedResult(
388388
attempts: outcome.attempts,
389389
artifactsDir: context.testArtifactsDir,
390390
error,
391-
...(attemptOutcome?.status === 'failed' && attemptOutcome.warnings
392-
? { warnings: [...attemptOutcome.warnings] }
391+
...(attemptOutcome?.status === 'failed' && (attemptOutcome.warnings?.length ?? 0) > 0
392+
? { warnings: [...attemptOutcome.warnings!] }
393393
: {}),
394394
...(attemptOutcome?.status === 'failed' && attemptOutcome.infrastructure
395395
? { infrastructure: true as const }

‎src/cli/replay-test/reporters/__tests__/junit.test.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,38 @@ test('buildReplayJunitXml escapes tricky failure title/message and round-trips t
8080
assert.ok(failure.text?.startsWith(TRICKY_MESSAGE));
8181
});
8282

83+
test('buildReplayJunitXml carries accumulated warnings of a failed test into system-out (#2560)', () => {
84+
const suite: ReplaySuiteResult = {
85+
total: 1,
86+
executed: 1,
87+
passed: 0,
88+
failed: 1,
89+
skipped: 0,
90+
notRun: 0,
91+
durationMs: 1200,
92+
failures: [],
93+
tests: [
94+
{
95+
file: '/tmp/flows/login.yaml',
96+
title: 'sign-in',
97+
session: 'default',
98+
status: 'failed',
99+
durationMs: 1200,
100+
attempts: 1,
101+
warnings: ['Optional Maestro assertVisible skipped at line 3: sheet owns focus'],
102+
error: { code: 'COMMAND_FAILED', message: 'tapOn Save did not resolve' },
103+
},
104+
],
105+
};
106+
suite.failures = suite.tests.filter((result) => result.status === 'failed');
107+
108+
const nodes = writeSuiteAndParse(suite);
109+
const testcase = findChild(findChild(nodes[0]!, 'testsuite')!, 'testcase');
110+
assert.ok(testcase);
111+
const systemOut = findChild(testcase, 'system-out');
112+
assert.ok(systemOut?.text?.includes('warning: Optional Maestro assertVisible skipped at line 3'));
113+
});
114+
83115
test('buildReplayJunitXml escapes tricky skip message', () => {
84116
const suite: ReplaySuiteResult = {
85117
total: 1,

‎src/commands/output/error.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,31 @@ test('printHumanError renders a compact divergence report unconditionally (not g
7676
// Not gated behind --debug: showDetails defaults to false/undefined here.
7777
});
7878

79+
// --- #2560: run-level warnings ride the failed response to the human surface ---
80+
81+
test('printHumanError renders run-level warnings carried in error details', async () => {
82+
const err = new AppError('REPLAY_DIVERGENCE', 'Replay failed at step 2 (tapOn "Save")', {
83+
step: 2,
84+
warnings: ['Optional Maestro assertVisible skipped at line 1: no match', 7],
85+
});
86+
87+
const output = await captureStderr(() => printHumanError(err));
88+
89+
assert.match(
90+
output,
91+
/Error \(REPLAY_DIVERGENCE\)[\s\S]*^Warning: Optional Maestro assertVisible skipped at line 1: no match$/m,
92+
);
93+
assert.equal(output.includes('Warning: 7'), false);
94+
});
95+
96+
test('printHumanError prints no Warning lines without a warnings channel', async () => {
97+
const err = new AppError('COMMAND_FAILED', 'tap failed');
98+
99+
const output = await captureStderr(() => printHumanError(err));
100+
101+
assert.equal(output.includes('Warning:'), false);
102+
});
103+
79104
// --- #1597: AMBIGUOUS_MATCH candidates print unconditionally, capped at 5 ---
80105

81106
test('printHumanError lists AMBIGUOUS_MATCH candidates unconditionally, not gated behind --debug', async () => {

‎src/commands/output/error.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import {
66
type NormalizedError,
77
} from '@agent-device/kernel/errors';
88
import { formatReplayDivergenceReport } from '@agent-device/ad-replay/divergence';
9+
import { collapseWarningText } from '../output-common.ts';
910

1011
export function printHumanError(
1112
err: AppError | NormalizedError,
@@ -20,6 +21,12 @@ export function printHumanError(
2021
if (normalized.hint) {
2122
process.stderr.write(`Hint: ${normalized.hint}\n`);
2223
}
24+
// Composable warnings (skipped `optional` steps, capture degradations) describe the run,
25+
// not this step's cause, so they ride at error level and must reach the reader of a
26+
// failed run too — a warning-only trace is otherwise invisible outside --json (#2560).
27+
for (const warning of readResponseWarnings(normalized.details)) {
28+
process.stderr.write(`Warning: ${collapseWarningText(warning)}\n`);
29+
}
2330
const candidateLines = formatErrorCandidateViews(readErrorCandidateViews(normalized.details));
2431
if (candidateLines.length > 0) {
2532
process.stderr.write(`${candidateLines.join('\n')}\n`);
@@ -47,6 +54,13 @@ export function printHumanError(
4754
}
4855
}
4956

57+
function readResponseWarnings(details: Record<string, unknown> | undefined): string[] {
58+
const warnings = details?.warnings;
59+
return Array.isArray(warnings)
60+
? warnings.filter((warning): warning is string => typeof warning === 'string')
61+
: [];
62+
}
63+
5064
function formatErrorCandidateViews(views: ErrorCandidateView[]): string[] {
5165
return views.flatMap((view) => {
5266
if (view.kind === 'element-match') {

‎src/daemon/replay/internal/__tests__/session-replay-maestro-failure.test.ts‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -96,11 +96,9 @@ test('typed Maestro failure response carries warnings accumulated before the fai
9696
{ warnings: ['Optional Maestro assertVisible skipped at line 2: no match'] },
9797
);
9898

99+
if (!response.error.details) throw new Error('expected error details on the failure response');
99100
const details = response.error.details;
100-
expect(details).toBeDefined();
101-
expect(details.warnings).toEqual([
102-
'Optional Maestro assertVisible skipped at line 2: no match',
103-
]);
101+
expect(details.warnings).toEqual(['Optional Maestro assertVisible skipped at line 2: no match']);
104102
expect((details.divergence as { kind: string }).kind).toBe('action-failure');
105103
});
106104

‎src/daemon/replay/internal/session-replay-runtime-failure-response.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,9 @@ export function buildReplayDivergenceFailureResponseFromDescriptor(params: {
104104
positionals,
105105
artifactPaths,
106106
...(snapshotDiagnostics ? { snapshotDiagnostics } : {}),
107-
...(warnings && warnings.length > 0 ? { warnings: [...warnings] } : {}),
107+
...(warnings && warnings.length > 0
108+
? { warnings: warnings.map((warning) => scrubReplayVarValues(warning, scrubVars)) }
109+
: {}),
108110
divergence,
109111
},
110112
},

‎website/docs/docs/replay-e2e.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ agent-device test ./workflows --reporter default --reporter junit:./tmp/junit.xm
123123
- `replay-timing.ndjson` records attempt, cleanup, and per-step start/stop events with durations. Upload it from CI even for passing runs when comparing local and CI performance.
124124
- Timeouts are cooperative: the runner marks the attempt failed at the timeout boundary, then gives the underlying replay a short grace period to stop before session cleanup.
125125
- The default text reporter streams live progress on stderr while a suite runs, then prints the final summary, failed tests, and passed-on-retry flaky tests. Use `--verbose` to include step traces in completed-test progress output.
126-
- The default reporter prints a `Warnings:` section after the summary when any test accumulated composable warnings — for example a Maestro step with `optional: true` that was skipped — whether that test passed or failed. `--json` carries the same strings in each test result's `warnings` array.
126+
- The default reporter prints a `Warnings:` section after the summary when any test accumulated composable warnings — for example a Maestro step with `optional: true` that was skipped — whether that test passed or failed. A failing `replay` run repeats the warnings it accumulated as `Warning:` lines after the error. `--json` carries the same strings in each test result's `warnings` array.
127127
- `--reporter` is repeatable. Built-ins are `default` for the console summary and `junit:<path>` for JUnit XML. Passing any explicit reporter list replaces the implicit default reporter, so include `--reporter default` when you also want terminal output. `--report-junit <path>` remains a compatibility alias for `--reporter junit:<path>`.
128128
- JUnit reports preserve legal Unicode and whitespace, and replace characters forbidden by XML 1.0 (such as terminal ESC or NUL) with `U+FFFD` (`�`) so CI parsers can read the report. JSON and other reporters retain the original suite values.
129129
- When `--fail-fast` and retries are both set, the current test still consumes its retries before the suite stops.

0 commit comments

Comments
 (0)