Skip to content

Commit bc79e05

Browse files
committed
fix(replay): carry optional-skip warnings to every human surface and name system surfaces in refusals
An optional: true Maestro step that skipped stayed a warning that the terminal never printed once a later step failed: the accumulated channel rode only on the success response, so the repro run in #2560 reported a bare failure with no trace of the skip. The engine now emits warnings on the failure event, the daemon copies them onto the failure response's error details, and the failed test result carries the same array the passed result already had; replay prints them through its facet formatter and the test reporter adds a Warnings section that renders whether tests pass or fail. A regular iOS presentation refusal from the runner now also carries the registry identity of the system surface the tree was acquired from in error details, so a capture-boundary miss under optional: true stops looking like a selector problem; the sparse-declared verdict path is owned by #2572.
1 parent 775163b commit bc79e05

28 files changed

Lines changed: 514 additions & 27 deletions

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,19 @@
1616
exactly as before, and a payload the runner did not declare sparse keeps the plain invariant byte for
1717
byte.
1818

19+
- Fixed: skipped `optional: true` Maestro steps are no longer silently reported as success on the
20+
human surface (#2560). A skip stays a warning that now travels with the run whether it later
21+
passes or fails: `replay` prints a `Warning:` line after its summary, `test` prints a
22+
`Warnings:` section after the suite summary naming each test, and a failed test result carries
23+
the same strings in its `warnings` array that `--json` and JUnit already had. The skip warning
24+
also appends the skipped failure's hint, so guidance that existed only in `--json` error details
25+
reaches the terminal.
26+
- Fixed: an iOS regular-presentation refusal ("regular iOS snapshot presentation requires a valid
27+
viewport") now carries a hint pointing at `snapshot --raw`, and when the acquired tree came from
28+
a system surface (e.g. a web sign-in sheet out of `SafariViewService`) the hint names that
29+
surface and the error details carry its registry identity (#2560). Previously a selector-backed
30+
command failing this way under `optional: true` gave no hint that the capture boundary — not the
31+
app — caused the miss.
1932
- Changed: a capture that a backend cut at one of its limits now says so in the snapshot's
2033
warnings, on every platform, instead of only setting `truncated: true` in JSON. The text path
2134
had no disclosure at all, so an agent read a screen missing its footer, tab bar, or the items

‎packages/capture-kit/src/ios-snapshot-engine/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,5 @@ export {
77
export { collectIosStructuralIdentifierSuppression } from './noise-structural.ts';
88
export { findNearestScrollableContainer, mergeReplacement, updateReplacement } from './tree.ts';
99
export { IosSnapshotEngineError, toIosSnapshotEngineErrorDetails } from './types.ts';
10+
export { iosViewportPresentationHint } from './invariants.ts';
1011
export type { SnapshotTreeRuleContext } from './tree.ts';
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import { IOS_VIEWPORT_PRESENTATION_HINT, iosViewportPresentationHint } from './invariants.ts';
4+
5+
test('viewport refusals hint at the raw projection on both presentation paths', () => {
6+
for (const reason of ['missing-viewport', 'invalid-viewport'] as const) {
7+
assert.equal(iosViewportPresentationHint(reason), IOS_VIEWPORT_PRESENTATION_HINT);
8+
assert.match(IOS_VIEWPORT_PRESENTATION_HINT, /snapshot --raw/);
9+
}
10+
});
11+
12+
test('non-viewport engine refusals get no viewport hint', () => {
13+
assert.equal(iosViewportPresentationHint('invalid-quality-payload'), undefined);
14+
assert.equal(iosViewportPresentationHint('projection-mismatch'), undefined);
15+
});

‎packages/capture-kit/src/ios-snapshot-engine/invariants.ts‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,21 @@ import { containsPoint, rectContains, isPositiveFiniteRect } from '@agent-device
77
import { normalizeType } from '@agent-device/contracts/snapshot';
88
import { validateIosSnapshotGraph } from './graph.ts';
99
import { IosSnapshotEngineError } from './types.ts';
10-
import type { IosSnapshotFoldPolicy } from './types.ts';
10+
import type { IosSnapshotEngineFailureReason, IosSnapshotFoldPolicy } from './types.ts';
1111

1212
const SCROLL_CONTAINER_TYPES = new Set(['collectionview', 'scrollview', 'table']);
1313

14+
export const IOS_VIEWPORT_PRESENTATION_HINT =
15+
'Use snapshot --raw to inspect the acquired iOS tree; regular presentation requires valid viewport evidence.';
16+
17+
export function iosViewportPresentationHint(
18+
reason: IosSnapshotEngineFailureReason,
19+
): string | undefined {
20+
return reason === 'missing-viewport' || reason === 'invalid-viewport'
21+
? IOS_VIEWPORT_PRESENTATION_HINT
22+
: undefined;
23+
}
24+
1425
export function resolveIosViewport(acquisition: IosSnapshotAcquisition): Rect {
1526
return resolveViewportEvidence(acquisition.viewport);
1627
}

‎packages/capture-kit/src/snapshot/ios-snapshot-runtime.ts‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import {
22
IosSnapshotEngineError,
3+
iosViewportPresentationHint,
34
publishIosSnapshot,
45
toIosSnapshotEngineErrorDetails,
56
} from '../ios-snapshot-engine/index.ts';
@@ -100,10 +101,7 @@ function snapshotWarnings(residue: readonly IosAcquisitionResidue[]): { warnings
100101
function throwIosSnapshotPresentationError(error: unknown): never {
101102
if (!(error instanceof IosSnapshotEngineError)) throw error;
102103
const details = toIosSnapshotEngineErrorDetails(error);
103-
const hint =
104-
error.reason === 'missing-viewport' || error.reason === 'invalid-viewport'
105-
? 'Use snapshot --raw to inspect the acquired iOS tree; regular presentation requires valid viewport evidence.'
106-
: undefined;
104+
const hint = iosViewportPresentationHint(error.reason);
107105
throw new AppError(
108106
'COMMAND_FAILED',
109107
error.message,

‎packages/contracts/src/replay.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ export type ReplaySuiteTestFailed = {
108108
attempts: number;
109109
artifactsDir?: string;
110110
error: DaemonError;
111+
/** Warnings accumulated before the failing step (skipped `optional` steps, capture degradations). */
112+
warnings?: string[];
111113
/** Present when the owning runtime classified the failure as device/runner infrastructure. */
112114
infrastructure?: true;
113115
shardIndex?: number;

‎packages/maestro/src/internal/__tests__/engine.test.ts‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,62 @@ describe('executeMaestroProgram', () => {
334334
expect(observer.commandCompleted).toHaveBeenCalledOnce();
335335
});
336336

337+
test('failure events carry warnings accumulated before the failing step (#2560)', async () => {
338+
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
339+
request.invalidateObservation();
340+
throw new AppError('COMMAND_FAILED', 'leaf command failed');
341+
});
342+
const port = makePort({
343+
observe: vi.fn(async ({ generation }) => ({ generation, matched: false })),
344+
execute,
345+
});
346+
const observer = { commandFailed: vi.fn() };
347+
const program = parseMaestroProgram(
348+
[
349+
'---',
350+
'- assertVisible:',
351+
' text: Missing assertion',
352+
' optional: true',
353+
'- tapOn: Missing target',
354+
].join('\n'),
355+
);
356+
357+
await expect(executeMaestroProgram(program, port, { observer })).rejects.toThrow(
358+
'leaf command failed',
359+
);
360+
361+
expect(observer.commandFailed).toHaveBeenCalledWith(
362+
expect.objectContaining({
363+
command: expect.objectContaining({ kind: 'tapOn' }),
364+
warnings: [expect.stringMatching(/Optional Maestro assertVisible skipped at line 2/)],
365+
}),
366+
);
367+
});
368+
369+
test('optional-skip warnings append the failure hint', async () => {
370+
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
371+
request.invalidateObservation();
372+
throw maestroTestFailure('Maestro target did not resolve to a visible element.', {
373+
hint: 'Regular presentation requires valid viewport evidence.',
374+
});
375+
});
376+
const port = makePort({
377+
observe: vi.fn(async ({ generation }) => ({ generation, matched: false })),
378+
execute,
379+
});
380+
const program = parseMaestroProgram(
381+
['---', '- tapOn:', ' text: Missing target', ' optional: true'].join('\n'),
382+
);
383+
384+
const result = await executeMaestroProgram(program, port);
385+
386+
expect(result.warnings).toEqual([
387+
expect.stringMatching(
388+
/Optional Maestro tapOn skipped at line 2: .*\(Regular presentation requires valid viewport evidence\.\)/,
389+
),
390+
]);
391+
});
392+
337393
test('observer failure cannot mask nested leaf failure provenance', async () => {
338394
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
339395
request.invalidateObservation();

‎packages/maestro/src/internal/engine-types.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,8 @@ export type MaestroEngineObserver = {
138138
runtimeMetrics?: MaestroRuntimeMetrics;
139139
error: unknown;
140140
artifactPaths: readonly string[];
141+
/** Warnings accumulated before this failure, including skipped `optional` steps. */
142+
warnings: readonly string[];
141143
},
142144
): void;
143145
};

‎packages/maestro/src/internal/facade-execution.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ export type MaestroFailedAction = MaestroActionEvent & {
5959
readonly runtimeMetrics?: MaestroCompletedActionEvent['runtimeMetrics'];
6060
readonly error: unknown;
6161
readonly artifactPaths: readonly string[];
62+
/** Warnings accumulated before this failure, including skipped `optional` steps. */
63+
readonly warnings: readonly string[];
6264
readonly isControl: boolean;
6365
readonly redactions: readonly { name: string; value: string }[];
6466
readonly resume:
@@ -212,6 +214,7 @@ function createObserver(
212214
durationMs: event.durationMs,
213215
error: event.error,
214216
artifactPaths: event.artifactPaths,
217+
warnings: [...event.warnings],
215218
isControl: isMaestroControlCommandDescriptor(event.command),
216219
redactions:
217220
event.command.kind === 'inputText' && event.command.text.length > 0

‎packages/maestro/src/internal/replay-plan-execution.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ async function executeObservedStep(
9696
...runtimeMetricsDelta(metricsBefore, state.port.readMetrics?.()),
9797
error: failure.error,
9898
artifactPaths: [...state.artifacts],
99+
warnings: [...state.warnings],
99100
}),
100101
);
101102
throw failure;

0 commit comments

Comments
 (0)