Skip to content

Commit 62e2dcc

Browse files
committed
fix(replay): carry optional-skip warnings to every human surface and name system surfaces in refusals
An optional: true Maestro step that skipped stayed a warning that the terminal never printed once a later step failed: the accumulated channel rode only on the success response, so the repro run in #2560 reported a bare failure with no trace of the skip. The engine now emits warnings on the failure event, the daemon copies them onto the failure response's error details, and the failed test result carries the same array the passed result already had; replay prints them through its facet formatter and the test reporter adds a Warnings section that renders whether tests pass or fail. A regular iOS presentation refusal from the runner now also carries the registry identity of the system surface the tree was acquired from in error details, so a capture-boundary miss under optional: true stops looking like a selector problem; the sparse-declared verdict path is owned by #2572.
1 parent 23fd3c9 commit 62e2dcc

24 files changed

Lines changed: 489 additions & 22 deletions

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,19 @@
9999
exactly as before, and a payload the runner did not declare sparse keeps the plain invariant byte for
100100
byte.
101101

102+
- Fixed: skipped `optional: true` Maestro steps are no longer invisible on the human surface
103+
(#2560). The warning a skip leaves now travels with the run whether it later passes or fails:
104+
`replay` prints a `Warning:` line after its summary and repeats the run's warnings after a
105+
failed run's error, `test` prints a `Warnings:` section after the suite summary naming each
106+
test, and a failed test result gained the `warnings` array the passing result already had, so
107+
`--json` and JUnit carry the skipped steps of a failing test too.
108+
- Fixed: an Apple runner presentation refusal now carries the registry identity of the system
109+
surface the tree was acquired from as `error.details.systemSurface` (#2560). Previously a
110+
selector-backed command failing at the capture boundary under `optional: true` gave nothing
111+
naming that boundary — the web sign-in sheet out of `SafariViewService` was invisible in the
112+
error, and the miss looked like a selector problem inside the app. The sparse-declared case
113+
already names the surface host in its hint via #2572; this covers the provenance everywhere
114+
the runner reports one.
102115
- Changed: a capture that a backend cut at one of its limits now says so in the snapshot's
103116
warnings, on every platform, instead of only setting `truncated: true` in JSON. The text path
104117
had no disclosure at all, so an agent read a screen missing its footer, tab bar, or the items

‎packages/contracts/src/replay.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ export type ReplaySuiteTestFailed = {
108108
attempts: number;
109109
artifactsDir?: string;
110110
error: DaemonError;
111+
/** Warnings accumulated before the failing step (skipped `optional` steps, capture degradations). */
112+
warnings?: string[];
111113
/** Present when the owning runtime classified the failure as device/runner infrastructure. */
112114
infrastructure?: true;
113115
shardIndex?: number;

‎packages/maestro/src/internal/__tests__/engine.test.ts‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,62 @@ describe('executeMaestroProgram', () => {
334334
expect(observer.commandCompleted).toHaveBeenCalledOnce();
335335
});
336336

337+
test('failure events carry warnings accumulated before the failing step (#2560)', async () => {
338+
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
339+
request.invalidateObservation();
340+
throw new AppError('COMMAND_FAILED', 'leaf command failed');
341+
});
342+
const port = makePort({
343+
observe: vi.fn(async ({ generation }) => ({ generation, matched: false })),
344+
execute,
345+
});
346+
const observer = { commandFailed: vi.fn() };
347+
const program = parseMaestroProgram(
348+
[
349+
'---',
350+
'- assertVisible:',
351+
' text: Missing assertion',
352+
' optional: true',
353+
'- tapOn: Missing target',
354+
].join('\n'),
355+
);
356+
357+
await expect(executeMaestroProgram(program, port, { observer })).rejects.toThrow(
358+
'leaf command failed',
359+
);
360+
361+
expect(observer.commandFailed).toHaveBeenCalledWith(
362+
expect.objectContaining({
363+
command: expect.objectContaining({ kind: 'tapOn' }),
364+
warnings: [expect.stringMatching(/Optional Maestro assertVisible skipped at line 2/)],
365+
}),
366+
);
367+
});
368+
369+
test('optional-skip warnings append the failure hint', async () => {
370+
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
371+
request.invalidateObservation();
372+
throw maestroTestFailure('Maestro target did not resolve to a visible element.', {
373+
hint: 'Regular presentation requires valid viewport evidence.',
374+
});
375+
});
376+
const port = makePort({
377+
observe: vi.fn(async ({ generation }) => ({ generation, matched: false })),
378+
execute,
379+
});
380+
const program = parseMaestroProgram(
381+
['---', '- tapOn:', ' text: Missing target', ' optional: true'].join('\n'),
382+
);
383+
384+
const result = await executeMaestroProgram(program, port);
385+
386+
expect(result.warnings).toEqual([
387+
expect.stringMatching(
388+
/Optional Maestro tapOn skipped at line 2: .*\(Regular presentation requires valid viewport evidence\.\)/,
389+
),
390+
]);
391+
});
392+
337393
test('observer failure cannot mask nested leaf failure provenance', async () => {
338394
const execute = vi.fn(async (request: MaestroRuntimeRequest) => {
339395
request.invalidateObservation();

‎packages/maestro/src/internal/engine-types.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,8 @@ export type MaestroEngineObserver = {
138138
runtimeMetrics?: MaestroRuntimeMetrics;
139139
error: unknown;
140140
artifactPaths: readonly string[];
141+
/** Warnings accumulated before this failure, including skipped `optional` steps. */
142+
warnings: readonly string[];
141143
},
142144
): void;
143145
};

‎packages/maestro/src/internal/facade-execution.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ export type MaestroFailedAction = MaestroActionEvent & {
5959
readonly runtimeMetrics?: MaestroCompletedActionEvent['runtimeMetrics'];
6060
readonly error: unknown;
6161
readonly artifactPaths: readonly string[];
62+
/** Warnings accumulated before this failure, including skipped `optional` steps. */
63+
readonly warnings: readonly string[];
6264
readonly isControl: boolean;
6365
readonly redactions: readonly { name: string; value: string }[];
6466
readonly resume:
@@ -215,6 +217,7 @@ function createObserver(
215217
durationMs: event.durationMs,
216218
error: event.error,
217219
artifactPaths: event.artifactPaths,
220+
warnings: [...event.warnings],
218221
isControl: isMaestroControlCommandDescriptor(event.command),
219222
redactions:
220223
event.command.kind === 'inputText' && event.command.text.length > 0

‎packages/maestro/src/internal/replay-plan-execution.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ async function executeObservedStep(
9696
...runtimeMetricsDelta(metricsBefore, state.port.readMetrics?.()),
9797
error: failure.error,
9898
artifactPaths: [...state.artifacts],
99+
warnings: [...state.warnings],
99100
}),
100101
);
101102
throw failure;

‎packages/maestro/src/internal/replay-plan-step-execution.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,13 @@ async function executeOptionalCommand(
9999

100100
function formatOptionalWarning(command: MaestroRuntimeCommand, error: unknown): string {
101101
const source = `${command.source.path ? `${command.source.path}:` : ''}line ${command.source.line}`;
102-
const message = error instanceof AppError ? error.message : String(error);
103-
return `Optional Maestro ${command.kind} skipped at ${source}: ${message}`;
102+
const detail =
103+
error instanceof AppError
104+
? typeof error.details?.hint === 'string'
105+
? `${error.message} (${error.details.hint})`
106+
: error.message
107+
: String(error);
108+
return `Optional Maestro ${command.kind} skipped at ${source}: ${detail}`;
104109
}
105110

106111
function isOptionalCommand(command: MaestroRuntimeCommand): boolean {

‎packages/platform-apple/src/runner/__tests__/snapshot-presentation.test.ts‎

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
import { AppError } from '@agent-device/kernel/errors';
22
import assert from 'node:assert/strict';
33
import { test } from 'vitest';
4+
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
5+
import type { IosSystemSurfaceProvenance } from '@agent-device/contracts/ios-system-surface';
46
import type { AppleRunnerSnapshotResult } from '../snapshot-presentation.ts';
5-
import { presentAppleRunnerSnapshot } from '../snapshot-presentation.ts';
7+
import { presentAppleRunnerSnapshot, readAppleSnapshotResult } from '../snapshot-presentation.ts';
68

79
const NO_VIEWPORT_ROOT = { index: 0, type: 'Application', label: 'App' };
810

@@ -73,6 +75,10 @@ test('a sparse capture of a presented system surface names the surface host', ()
7375
String(error.details?.hint),
7476
/com\.apple\.SafariViewService hosts the surface presented over the app/,
7577
);
78+
assert.deepEqual(error.details?.systemSurface, {
79+
bundleId: 'com.apple.SafariViewService',
80+
kind: 'web-auth',
81+
});
7682
});
7783

7884
test('a sparse payload failing another invariant still carries the verdict', () => {
@@ -125,3 +131,61 @@ test('a sparse verdict still presents the nodes it did read', () => {
125131
['App', 'Not Now'],
126132
);
127133
});
134+
135+
const SYSTEM_SHEET: IosSystemSurfaceProvenance = {
136+
bundleId: 'com.apple.SafariViewService',
137+
kind: 'web-auth',
138+
};
139+
140+
test('a presented system surface travels with an undeclared-payload refusal as typed provenance', () => {
141+
try {
142+
presentAppleRunnerSnapshot('device-1', undefined, {
143+
nodes: [NO_VIEWPORT_ROOT],
144+
quality: { state: 'healthy', backend: 'tree' },
145+
systemSurface: SYSTEM_SHEET,
146+
});
147+
} catch (error) {
148+
assert.ok(error instanceof AppError);
149+
assert.deepEqual((error.details as Record<string, unknown>).systemSurface, SYSTEM_SHEET);
150+
assert.equal((error.details as Record<string, unknown>).snapshotQuality, undefined);
151+
return;
152+
}
153+
assert.fail('expected the presentation to refuse the payload');
154+
});
155+
156+
test('readAppleSnapshotResult keeps registry-known system surface provenance only', () => {
157+
const known = readAppleSnapshotResult({
158+
nodes: [],
159+
systemSurface: { bundleId: 'com.apple.SafariViewService', kind: 'web-auth' },
160+
});
161+
assert.deepEqual(known.systemSurface, SYSTEM_SHEET);
162+
163+
const unknown = readAppleSnapshotResult({
164+
nodes: [],
165+
systemSurface: { bundleId: 'com.example.PhishingService', kind: 'web-auth' },
166+
});
167+
assert.equal(unknown.systemSurface, undefined);
168+
});
169+
170+
test('a healthy payload with valid viewport roots still presents', () => {
171+
const screen: RawSnapshotNode = {
172+
index: 0,
173+
type: 'Application',
174+
rect: { x: 0, y: 0, width: 390, height: 844 },
175+
};
176+
const button: RawSnapshotNode = {
177+
index: 1,
178+
parentIndex: 0,
179+
type: 'Button',
180+
label: 'Not Now',
181+
rect: { x: 16, y: 400, width: 80, height: 32 },
182+
hittable: true,
183+
};
184+
const nodes = presentAppleRunnerSnapshot('device-1', undefined, {
185+
nodes: [screen, button],
186+
});
187+
assert.deepEqual(
188+
nodes.map((node) => node.index),
189+
[0, 1],
190+
);
191+
});

‎packages/platform-apple/src/runner/snapshot-presentation.ts‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ export function readAppleSnapshotResult(
4545
nodes: Array.isArray(result.nodes) ? (result.nodes as RawSnapshotNode[]) : undefined,
4646
truncated: typeof result.truncated === 'boolean' ? result.truncated : undefined,
4747
quality: readSnapshotQualityVerdict(result.snapshotQuality),
48-
qualityPayload: readQualityPayload(result.qualityPayload),
48+
qualityPayload: readQualityPayload(result.qualityPayload, systemSurface),
4949
runnerFatal: result.runnerFatal === true,
5050
...(systemSurface ? { systemSurface } : {}),
5151
message:
@@ -131,6 +131,7 @@ function throwSnapshotPresentationError(error: unknown, result: AppleRunnerSnaps
131131
error.message,
132132
{
133133
...toIosSnapshotEngineErrorDetails(error),
134+
...(result.systemSurface ? { systemSurface: result.systemSurface } : {}),
134135
snapshotQuality: {
135136
state: verdict.state,
136137
backend: verdict.backend,
@@ -142,7 +143,7 @@ function throwSnapshotPresentationError(error: unknown, result: AppleRunnerSnaps
142143
error,
143144
);
144145
}
145-
throwSnapshotEngineError(error);
146+
throwSnapshotEngineError(error, result.systemSurface);
146147
}
147148

148149
function sparseCaptureHint(
@@ -159,14 +160,18 @@ function sparseCaptureHint(
159160
.join(' ');
160161
}
161162

162-
function readQualityPayload(value: unknown): IosRunnerQualityPayloadFacts | undefined {
163+
function readQualityPayload(
164+
value: unknown,
165+
systemSurface?: IosSystemSurfaceProvenance,
166+
): IosRunnerQualityPayloadFacts | undefined {
163167
if (value === undefined) return undefined;
164168
if (!isRecord(value) || !Array.isArray(value.nodes) || typeof value.truncated !== 'boolean') {
165169
throwSnapshotEngineError(
166170
new IosSnapshotEngineError(
167171
'invalid-quality-payload',
168172
'iOS runner returned an invalid quality payload',
169173
),
174+
systemSurface,
170175
);
171176
}
172177
if (value.scope !== undefined && value.scope !== null) {
@@ -176,6 +181,7 @@ function readQualityPayload(value: unknown): IosRunnerQualityPayloadFacts | unde
176181
'iOS runner quality payload must be unscoped',
177182
{ field: 'scope' },
178183
),
184+
systemSurface,
179185
);
180186
}
181187
return { nodes: value.nodes as RawSnapshotNode[], truncated: value.truncated, scope: null };
@@ -204,12 +210,18 @@ function isRecord(value: unknown): value is Record<string, unknown> {
204210
return typeof value === 'object' && value !== null && !Array.isArray(value);
205211
}
206212

207-
function throwSnapshotEngineError(error: unknown): never {
213+
function throwSnapshotEngineError(
214+
error: unknown,
215+
systemSurface?: IosSystemSurfaceProvenance,
216+
): never {
208217
if (!(error instanceof IosSnapshotEngineError)) throw error;
209218
throw new AppError(
210219
'COMMAND_FAILED',
211220
error.message,
212-
toIosSnapshotEngineErrorDetails(error),
221+
{
222+
...toIosSnapshotEngineErrorDetails(error),
223+
...(systemSurface ? { systemSurface } : {}),
224+
},
213225
error,
214226
);
215227
}

‎packages/replay-test/src/internal/session-test-attempt.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,9 @@ function buildReplayTestFailedResult(
388388
attempts: outcome.attempts,
389389
artifactsDir: context.testArtifactsDir,
390390
error,
391+
...(attemptOutcome?.status === 'failed' && attemptOutcome.warnings
392+
? { warnings: [...attemptOutcome.warnings] }
393+
: {}),
391394
...(attemptOutcome?.status === 'failed' && attemptOutcome.infrastructure
392395
? { infrastructure: true as const }
393396
: {}),

0 commit comments

Comments
 (0)