Skip to content

Commit 5fb2860

Browse files
authored
fix: carry response warnings to every surface, and stamp the sheet behind a refusal (#2571)
* fix(replay): carry optional-skip warnings to every human surface and name system surfaces in refusals An optional: true Maestro step that skipped stayed a warning that the terminal never printed once a later step failed: the accumulated channel rode only on the success response, so the repro run in #2560 reported a bare failure with no trace of the skip. The engine now emits warnings on the failure event, the daemon copies them onto the failure response's error details, and the failed test result carries the same array the passed result already had; replay prints them through its facet formatter and the test reporter adds a Warnings section that renders whether tests pass or fail. A regular iOS presentation refusal from the runner now also carries the registry identity of the system surface the tree was acquired from in error details, so a capture-boundary miss under optional: true stops looking like a selector problem; the sparse-declared verdict path is owned by #2572. * fix(replay): surface warnings on failed runs and keep them honest A failing `replay` still printed no trace of the run's accumulated warnings: the human error renderer read message, hint, and divergence but never the error-level warnings channel, so the pass/fail asymmetry survived on that surface. It now prints each warning after the error. Round-two corrections: warnings on the divergence response are scrubbed against replay vars like message and hint already are; a failed attempt copies warnings only when non-empty, matching the passing branch; the optional-skip hint append is removed because no producer of a maestro test failure emits a hint — capture-error lineage stays with #2450 — along with the CHANGELOG claim it rested on; the failed-result copy gains its own module test, and JUnit covers a failed test's warning lines. * refactor(replay-test): make an attempt failure's warnings channel always present An empty array and an absent key said the same thing two ways, so every consumer needed both guards. The failed outcome now always carries the warnings array, like the passed outcome already did, and the result builder narrows the attempt once instead of per field. This also removes the clone the shared candidate-view formatter had grown between the CLI and MCP error renderers — the MCP error text gained the same warnings lines the CLI stderr prints. * refactor(kernel): own the response-warnings field contract beside the message reader Four call sites read the composable warnings field with their own Array.isArray filters (the daemon append, the daemon attempt outcome, the CLI success line, the CLI/MCP error text), and the PR had grown a fifth. The field has one contract now: kernel/success-text readResponseWarnings, next to readCommandMessage, with the daemon append, the attempt outcome, messageWithWarningsText, and both error renderers as its named consumers. The CLI-side reader the PR introduced is deleted, not wrapped.
1 parent e3cbc91 commit 5fb2860

43 files changed

Lines changed: 747 additions & 80 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,19 @@
139139
exactly as before, and a payload the runner did not declare sparse keeps the plain invariant byte for
140140
byte.
141141

142+
- Fixed: skipped `optional: true` Maestro steps are no longer invisible on the human surface
143+
(#2560). The warning a skip leaves now travels with the run whether it later passes or fails:
144+
`replay` prints a `Warning:` line after its summary and repeats the run's warnings after a
145+
failed run's error, `test` prints a `Warnings:` section after the suite summary naming each
146+
test, and a failed test result gained the `warnings` array the passing result already had, so
147+
`--json` and JUnit carry the skipped steps of a failing test too.
148+
- Fixed: an Apple runner presentation refusal now carries the registry identity of the system
149+
surface the tree was acquired from as `error.details.systemSurface` (#2560). Previously a
150+
selector-backed command failing at the capture boundary under `optional: true` gave nothing
151+
naming that boundary — the web sign-in sheet out of `SafariViewService` was invisible in the
152+
error, and the miss looked like a selector problem inside the app. The sparse-declared case
153+
already names the surface host in its hint via #2572; this covers the provenance everywhere
154+
the runner reports one.
142155
- Changed: a capture that a backend cut at one of its limits now says so in the snapshot's
143156
warnings, on every platform, instead of only setting `truncated: true` in JSON. The text path
144157
had no disclosure at all, so an agent read a screen missing its footer, tab bar, or the items

‎packages/contracts/src/replay.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ export type ReplaySuiteTestFailed = {
108108
attempts: number;
109109
artifactsDir?: string;
110110
error: DaemonError;
111+
/** Warnings accumulated before the failing step (skipped `optional` steps, capture degradations). */
112+
warnings?: string[];
111113
/** Present when the owning runtime classified the failure as device/runner infrastructure. */
112114
infrastructure?: true;
113115
shardIndex?: number;
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import { readResponseWarnings } from '@agent-device/kernel/success-text';
4+
import { readSerializedSnapshotCaptureAnnotations } from './snapshot-capture-annotations.ts';
5+
6+
test('the annotations filter and the shared warnings parser agree on adversarial arrays', () => {
7+
for (const warnings of [
8+
['a note'],
9+
['a note', 42, { nested: true }, null],
10+
['', 'kept'],
11+
['a note', ''],
12+
]) {
13+
assert.deepEqual(
14+
readSerializedSnapshotCaptureAnnotations({ warnings }).warnings,
15+
readResponseWarnings({ warnings }),
16+
`drift for ${JSON.stringify(warnings)}`,
17+
);
18+
}
19+
});
20+
21+
test('an empty warnings array serializes back to absent', () => {
22+
assert.equal(readSerializedSnapshotCaptureAnnotations({ warnings: [] }).warnings, undefined);
23+
});
24+
25+
test('absent or non-array warnings stay absent on the serialized annotations', () => {
26+
assert.equal(readSerializedSnapshotCaptureAnnotations({}).warnings, undefined);
27+
assert.equal(
28+
readSerializedSnapshotCaptureAnnotations({ warnings: 'a note' }).warnings,
29+
undefined,
30+
);
31+
});

‎packages/contracts/src/snapshot-capture-annotations.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,9 @@ export function readSerializedSnapshotCaptureAnnotations(
5757
data: Record<string, unknown>,
5858
): PublicSnapshotCaptureAnnotations {
5959
const androidSnapshot = readObject(data.androidSnapshot);
60+
// Declared exception to kernel's shared `readResponseWarnings` (see its doc): this facade
61+
// pins its eager module closure, and absent-or-non-array keeps the serialized tri-state.
62+
// `snapshot-capture-annotations.test.ts` cross-checks this filter against the shared parser.
6063
const warnings = Array.isArray(data.warnings)
6164
? data.warnings.filter((entry): entry is string => typeof entry === 'string')
6265
: undefined;
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import assert from 'node:assert/strict';
2+
import { describe, test } from 'vitest';
3+
import { readCommandMessage } from './success-text.ts';
4+
5+
describe('readCommandMessage', () => {
6+
test('an empty message is absent, not an empty success line', () => {
7+
assert.equal(readCommandMessage({ message: '' }), null);
8+
assert.equal(readCommandMessage({ message: 42 }), null);
9+
assert.equal(readCommandMessage(undefined), null);
10+
assert.equal(readCommandMessage({ message: 'Replayed 7 steps' }), 'Replayed 7 steps');
11+
});
12+
});

‎packages/kernel/src/success-text.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,25 @@ export function withSuccessText<T extends Record<string, unknown>>(
1212
export function readCommandMessage(data: Record<string, unknown> | undefined): string | null {
1313
return typeof data?.message === 'string' && data.message.length > 0 ? data.message : null;
1414
}
15+
16+
/**
17+
* The composable response-warnings channel (skipped `optional` steps, capture
18+
* degradations): readers that project a response or error record onto note
19+
* strings go through here — daemon append, attempt outcome, and
20+
* recovered-quality latch, CLI success line, CLI/MCP error text, SDK client
21+
* (open and screenshot result), and the
22+
* snapshot text renderer — so one field contract has one parser. Consumers may
23+
* add rendering rules on top (snapshot text and screenshot result drop empty
24+
* notes; screenshot result keeps absent-means-undefined). The one declared
25+
* exception is contracts' `readSerializedSnapshotCaptureAnnotations`, which
26+
* keeps a local copy of the filter: contracts facades pin their eager module
27+
* closure (`scripts/__tests__/eager-closure-budgets.test.ts`) and this module
28+
* is outside it; its test cross-checks both parses so the contract cannot
29+
* drift. Non-string entries are other producers' bugs.
30+
*/
31+
export function readResponseWarnings(data: Record<string, unknown> | undefined): string[] {
32+
const warnings = data?.warnings;
33+
return Array.isArray(warnings)
34+
? warnings.filter((warning): warning is string => typeof warning === 'string')
35+
: [];
36+
}

‎packages/maestro/src/internal/__tests__/engine.test.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,38 @@ describe('executeMaestroProgram', () => {
334334
expect(observer.commandCompleted).toHaveBeenCalledOnce();
335335
});
336336

337+
test('failure events carry warnings accumulated before the failing step (#2560)', async () => {
338+
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
339+
request.invalidateObservation();
340+
throw new AppError('COMMAND_FAILED', 'leaf command failed');
341+
});
342+
const port = makePort({
343+
observe: vi.fn(async ({ generation }) => ({ generation, matched: false })),
344+
execute,
345+
});
346+
const observer = { commandFailed: vi.fn() };
347+
const program = parseMaestroProgram(
348+
[
349+
'---',
350+
'- assertVisible:',
351+
' text: Missing assertion',
352+
' optional: true',
353+
'- tapOn: Missing target',
354+
].join('\n'),
355+
);
356+
357+
await expect(executeMaestroProgram(program, port, { observer })).rejects.toThrow(
358+
'leaf command failed',
359+
);
360+
361+
expect(observer.commandFailed).toHaveBeenCalledWith(
362+
expect.objectContaining({
363+
command: expect.objectContaining({ kind: 'tapOn' }),
364+
warnings: [expect.stringMatching(/Optional Maestro assertVisible skipped at line 2/)],
365+
}),
366+
);
367+
});
368+
337369
test('observer failure cannot mask nested leaf failure provenance', async () => {
338370
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
339371
request.invalidateObservation();

‎packages/maestro/src/internal/engine-types.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,8 @@ export type MaestroEngineObserver = {
138138
runtimeMetrics?: MaestroRuntimeMetrics;
139139
error: unknown;
140140
artifactPaths: readonly string[];
141+
/** Warnings accumulated before this failure, including skipped `optional` steps. */
142+
warnings: readonly string[];
141143
},
142144
): void;
143145
};

‎packages/maestro/src/internal/facade-execution.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ export type MaestroFailedAction = MaestroActionEvent & {
5959
readonly runtimeMetrics?: MaestroCompletedActionEvent['runtimeMetrics'];
6060
readonly error: unknown;
6161
readonly artifactPaths: readonly string[];
62+
/** Warnings accumulated before this failure, including skipped `optional` steps. */
63+
readonly warnings: readonly string[];
6264
readonly isControl: boolean;
6365
readonly redactions: readonly { name: string; value: string }[];
6466
readonly resume:
@@ -215,6 +217,7 @@ function createObserver(
215217
durationMs: event.durationMs,
216218
error: event.error,
217219
artifactPaths: event.artifactPaths,
220+
warnings: [...event.warnings],
218221
isControl: isMaestroControlCommandDescriptor(event.command),
219222
redactions:
220223
event.command.kind === 'inputText' && event.command.text.length > 0

‎packages/maestro/src/internal/replay-plan-execution.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ async function executeObservedStep(
9696
...runtimeMetricsDelta(metricsBefore, state.port.readMetrics?.()),
9797
error: failure.error,
9898
artifactPaths: [...state.artifacts],
99+
warnings: [...state.warnings],
99100
}),
100101
);
101102
throw failure;

0 commit comments

Comments
 (0)