Skip to content

Commit 51d5906

Browse files
authored
fix(test): make the iOS simulator smoke lane's red/green signal meaningful (#2491) (#3336)
* test(ios): accept the disclosed runner fallback in the depth-frontier snapshot assertion (#3328) The lane asserted a regular snapshot must not carry snapshotQuality at all, but the product's contract is a disclosure, not an absence: the AX bridge publishes no verdict, and the XCTest runner the route falls back to always stamps its serving strategy. When the bridge probe circuit is open for the app generation (a legitimate cold/loaded-host state), the runner-served healthy 'tree' verdict made the lane red twice on main (runs 37145696803, 37459193672). The assertion now keys on the typed verdict: an absent (bridge) or healthy verdict is accepted, a degraded 'recovered' or 'sparse' verdict still fails, never on the warning string. * test(ios): re-issue a live-lane step whose typed verdict says observation was prevented (#2491) The replay steps in the iOS smoke job run with --retries 2 while the fixture-backed E2E step runs node --test with no retry layer, so one cold runner restart inside one 10s wait fails the whole job. Seven of the last ten main failures are this one class: a wait whose own error names wait_capture_stalled, wait_runner_restart_exhausted, or wait_readiness_exhausted and carries retriable: true — the product itself says 'retry' on those verdicts. The harness gains an opt-in per-step classifier; the iOS lane feeds it a predicate keyed on that typed conjunction only (wire retriable AND the wait taxonomy reason), never on error text. A re-attributed miss re-issues the step once; a readable miss (target absent, deadline exhausted after a readable capture, a wrong asserted value, a runner crash) stays a hard failure. Steps that already own a miss policy (allowFailure/expectFailure) are exempt so existing retry loops keep their semantics. * test(ios): reject a snapshot quality state outside the declared vocabulary The acquisition assertion only rejected 'sparse' and unexempt 'recovered', so a verdict with any other state (renamed enum, diverged runner) fell through both branches and certified depth facts off a tree the lane never classified. The state set is now pinned to the kernel-declared SNAPSHOT_QUALITY_STATES, with a negative test for states outside it.
1 parent d87cf26 commit 51d5906

7 files changed

Lines changed: 495 additions & 12 deletions
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
import assert from 'node:assert/strict';
2+
import test from 'node:test';
3+
4+
import { assertSimulatorSnapshotAcquisition } from './ios-simulator-e2e/live-snapshot-depth-frontier.ts';
5+
6+
// #3328, pinned without a simulator: the depth-frontier scenario accepts either acquisition it
7+
// can present depth facts from — the AX bridge (which publishes no verdict) or the XCTest runner
8+
// the route sent the capture to (which always stamps its strategy) — and rejects only a capture
9+
// whose own typed verdict says it degraded or served nothing. The positives are the exact shapes
10+
// of the two failing main runs (37145696803, 37459193672), whose healthy `tree` verdict the old
11+
// assertion failed on the word `tree`.
12+
13+
const HITTABLE = 'viewport-derived hittability evidence';
14+
15+
function capture(quality: unknown): { json?: any } {
16+
return {
17+
json: {
18+
success: true,
19+
data: {
20+
nodes: [{ index: 0, depth: 0, hittable: true }],
21+
...(quality === undefined ? {} : { snapshotQuality: quality }),
22+
warnings: [HITTABLE],
23+
},
24+
},
25+
};
26+
}
27+
28+
test('an AX-bridge capture carries no verdict and is accepted', () => {
29+
assertSimulatorSnapshotAcquisition(capture(undefined), 'regular depth-1 snapshot');
30+
});
31+
32+
for (const backend of ['tree', 'queries', 'private-ax'] as const) {
33+
test(`a runner-served capture disclosing the ${backend} strategy is accepted`, () => {
34+
assertSimulatorSnapshotAcquisition(
35+
capture({ state: 'healthy', backend, timing: { acquisitionMs: 64.7, presentationMs: 0.09 } }),
36+
'regular depth-1 snapshot',
37+
);
38+
});
39+
}
40+
41+
for (const reasonCode of ['deferred', 'requested-backend'] as const) {
42+
test(`a pre-selected-backend recovered verdict (${reasonCode}) is accepted`, () => {
43+
assertSimulatorSnapshotAcquisition(
44+
capture({ state: 'recovered', backend: 'private-ax', reasonCode }),
45+
'full raw visible-depth snapshot',
46+
);
47+
});
48+
}
49+
50+
test('a capture that recovered to another strategy mid-capture is rejected', () => {
51+
// The closest negative to the pre-selected pair: the same `recovered` state with a degradation
52+
// code means the strategy the presented depth belongs to failed mid-capture, so the tree the
53+
// frontier assertions would read is not a comparable view of the screen (#1569).
54+
for (const reasonCode of ['capture-failed', 'presentation-failed', undefined]) {
55+
assert.throws(
56+
() =>
57+
assertSimulatorSnapshotAcquisition(
58+
capture({ state: 'recovered', backend: 'tree', ...(reasonCode ? { reasonCode } : {}) }),
59+
'regular depth-1 snapshot',
60+
),
61+
/fell back to another capture strategy mid-capture/,
62+
);
63+
}
64+
});
65+
66+
test('a sparse capture is rejected whatever strategy it names', () => {
67+
assert.throws(
68+
() =>
69+
assertSimulatorSnapshotAcquisition(
70+
capture({ state: 'sparse', backend: 'tree', reasonCode: 'sparse-tree' }),
71+
'regular depth-1 snapshot',
72+
),
73+
/no backend served this screen/,
74+
);
75+
});
76+
77+
test('a verdict with a state outside the declared vocabulary is rejected', () => {
78+
// The kernel declares ['healthy', 'recovered', 'sparse'] and the runner's allCases is pinned to
79+
// the same fixture; a verdict outside that set means the two sides diverged, and certifying
80+
// depth facts off a tree the lane cannot classify is the same meaningless-green defect as
81+
// ruling a sparse capture healthy.
82+
for (const state of ['unknown', 'degraded', undefined]) {
83+
assert.throws(
84+
() =>
85+
assertSimulatorSnapshotAcquisition(
86+
capture({ ...(state === undefined ? {} : { state }), backend: 'tree' }),
87+
'regular depth-1 snapshot',
88+
),
89+
/state outside the declared vocabulary/,
90+
);
91+
}
92+
});
93+
94+
test('a capture that declares its hittability evidence missing is rejected', () => {
95+
assert.throws(
96+
() =>
97+
assertSimulatorSnapshotAcquisition(
98+
{
99+
json: {
100+
success: true,
101+
data: {
102+
nodes: [{ index: 0, depth: 0 }],
103+
warnings: [
104+
'iOS snapshot acquisition does not provide hittability evidence; regular snapshots omit unverified hittability while raw snapshots preserve supplied facts.',
105+
],
106+
},
107+
},
108+
},
109+
'regular depth-1 snapshot',
110+
),
111+
/must derive hittability/,
112+
);
113+
});
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
import assert from 'node:assert/strict';
2+
import test from 'node:test';
3+
4+
import { WAIT_REASONS } from '@agent-device/contracts/wait';
5+
import type { CliJsonResult } from './cli-json.ts';
6+
import { isObservationPreventedStepMiss } from './ios-simulator-e2e/live-step-retry-policy.ts';
7+
8+
// The #2491 retry policy for the iOS live lane, pinned without a simulator: which typed wire
9+
// verdicts the lane may re-issue one step for, and — the closest negatives — which look close but
10+
// must fail at once. The pairing of `retriable: true` with an observation-prevented
11+
// `details.reason` is the whole predicate; neither signal alone activates a re-issue.
12+
13+
function stepResult(json: unknown, status = 1): CliJsonResult {
14+
return { json, status, stdout: '', stderr: '' };
15+
}
16+
17+
function waitFailure(reason: string, retriable: boolean | undefined): CliJsonResult {
18+
return stepResult({
19+
error: {
20+
code: 'COMMAND_FAILED',
21+
details: { reason },
22+
...(retriable === undefined ? {} : { retriable }),
23+
},
24+
});
25+
}
26+
27+
for (const reason of [
28+
WAIT_REASONS.captureStalled,
29+
WAIT_REASONS.runnerRestartExhausted,
30+
WAIT_REASONS.readinessExhausted,
31+
]) {
32+
test(`${reason} with the product's retriable verdict re-issues the step`, () => {
33+
assert.equal(isObservationPreventedStepMiss(waitFailure(reason, true)), true);
34+
});
35+
}
36+
37+
test('an observation-prevented reason without the product retriable verdict does not re-issue', () => {
38+
// The negative that keeps the predicate keyed on the conjunction: a response carrying the same
39+
// reason but marked non-retriable (or carrying no verdict) must not buy a re-issue.
40+
assert.equal(
41+
isObservationPreventedStepMiss(waitFailure(WAIT_REASONS.captureStalled, false)),
42+
false,
43+
);
44+
assert.equal(
45+
isObservationPreventedStepMiss(waitFailure(WAIT_REASONS.captureStalled, undefined)),
46+
false,
47+
);
48+
});
49+
50+
test('a retriable target-absent or deadline-exceeded wait does not re-issue', () => {
51+
// The screen was readable: the target was not there, or a readable capture consumed the
52+
// remaining budget. Re-issuing would hide a real absence, so these stay hard failures.
53+
assert.equal(isObservationPreventedStepMiss(waitFailure(WAIT_REASONS.targetAbsent, true)), false);
54+
assert.equal(
55+
isObservationPreventedStepMiss(waitFailure(WAIT_REASONS.deadlineExceeded, true)),
56+
false,
57+
);
58+
});
59+
60+
test('a non-wait retriable failure does not re-issue', () => {
61+
// A retriable verdict on a response with no wait taxonomy reason (e.g. a leased-busy device on
62+
// a click) is not evidence that observation of THIS step was prevented.
63+
assert.equal(
64+
isObservationPreventedStepMiss(
65+
stepResult({ error: { code: 'DEVICE_IN_USE', retriable: true } }),
66+
),
67+
false,
68+
);
69+
});
70+
71+
test('a successful step result is never a re-attributable miss', () => {
72+
assert.equal(isObservationPreventedStepMiss(stepResult(undefined, 0)), false);
73+
assert.equal(isObservationPreventedStepMiss(stepResult(undefined)), false);
74+
});

‎test/integration/ios-simulator-e2e/live-harness.ts‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
} from '../live-device-e2e/runtime.ts';
1414
import type { IosSimulatorBehaviorId } from './behavior-coverage.ts';
1515
import { liveCommandsForScenario } from './coverage.ts';
16+
import { isObservationPreventedStepMiss } from './live-step-retry-policy.ts';
1617
import { liveBehaviorsForScenario, writeCoverageReport } from './live-coverage-report.ts';
1718

1819
export { assertCoverageComplete, writeCoverageReport } from './live-coverage-report.ts';
@@ -67,7 +68,26 @@ const harness = createLiveDeviceHarness<LiveContext, IosSimulatorBehaviorId>({
6768
writeCoverageReport,
6869
});
6970

70-
export const { runScenario, runStep, sessionExists, verifyBehavior, verifyCommand } = harness;
71+
/**
72+
* Every iOS live step carries the observation-prevented re-issue policy (#2491): one runner
73+
* restart or bridge rebuild re-issues its own wait instead of failing the job, while a readable
74+
* miss (target absent, budget exhausted after a readable capture, wrong asserted value) fails at
75+
* once exactly as before. Steps that already own a miss policy (`allowFailure` / `expectFailure`,
76+
* like the destination-wait and cleanup retry loops) are exempt inside the harness itself.
77+
*/
78+
export function runStep(
79+
context: LiveContext,
80+
step: string,
81+
args: string[],
82+
options: Parameters<typeof harness.runStep>[3] = {},
83+
): Promise<CliJsonResult> {
84+
return harness.runStep(context, step, args, {
85+
reattemptInfrastructureMiss: isObservationPreventedStepMiss,
86+
...options,
87+
});
88+
}
89+
90+
export const { runScenario, sessionExists, verifyBehavior, verifyCommand } = harness;
7191

7292
export function verifyNestedReplayCommand(
7393
context: LiveContext,

‎test/integration/ios-simulator-e2e/live-snapshot-depth-frontier.ts‎

Lines changed: 59 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
import assert from 'node:assert/strict';
22

3+
import { SNAPSHOT_QUALITY_STATES } from '@agent-device/kernel/snapshot';
4+
35
import {
46
assertWaitText,
57
type LiveSnapshotNode as SnapshotNode,
@@ -31,7 +33,7 @@ export async function assertRegularVisibleDepthFrontier(context: LiveContext): P
3133
'1',
3234
'--debug',
3335
]);
34-
assertSimulatorBridgeSnapshot(regular, 'regular depth-1 snapshot');
36+
assertSimulatorSnapshotAcquisition(regular, 'regular depth-1 snapshot');
3537
const regularNodes = snapshotNodes(regular);
3638
const regularRoot = requireRoot(regularNodes, 'regular depth-1 snapshot');
3739
const projectedChild = requireIdentifier(regularNodes, CHILD_ID, 'regular depth-1 snapshot');
@@ -59,7 +61,7 @@ export async function assertRegularVisibleDepthFrontier(context: LiveContext): P
5961
'snapshot',
6062
'--raw',
6163
]);
62-
assertSimulatorBridgeSnapshot(rawFull, 'full raw visible-depth snapshot');
64+
assertSimulatorSnapshotAcquisition(rawFull, 'full raw visible-depth snapshot');
6365
const rawFullNodes = snapshotNodes(rawFull);
6466
const rawChild = requireIdentifier(rawFullNodes, CHILD_ID, 'full raw visible-depth snapshot');
6567
assert.ok(
@@ -73,7 +75,7 @@ export async function assertRegularVisibleDepthFrontier(context: LiveContext): P
7375
'--depth',
7476
'1',
7577
]);
76-
assertSimulatorBridgeSnapshot(rawDepthOne, 'raw depth-1 visible-depth snapshot');
78+
assertSimulatorSnapshotAcquisition(rawDepthOne, 'raw depth-1 visible-depth snapshot');
7779
const rawDepthOneNodes = snapshotNodes(rawDepthOne);
7880
assert.equal(
7981
rawDepthOneNodes.some((node) => node.identifier === CHILD_ID),
@@ -122,12 +124,60 @@ function numericDepth(node: SnapshotNode): number {
122124
return node.depth as number;
123125
}
124126

125-
function assertSimulatorBridgeSnapshot(result: { json?: any }, description: string): void {
126-
assert.equal(
127-
result.json?.data?.snapshotQuality?.backend,
128-
undefined,
129-
`${description} must not carry XCTest tree quality metadata: ${JSON.stringify(result)}`,
130-
);
127+
/**
128+
* Reason codes that name a PRE-SELECTED backend rather than a capture that degraded
129+
* (`deferred`: the runner's penalty circuit; `requested-backend`: the caller asked for that
130+
* strategy). They are the two codes the product's own quality-warning renderer exempts from any
131+
* degradation sentence for the same reason: nothing on THIS capture went wrong.
132+
*/
133+
const PRE_SELECTED_REASON_CODES: ReadonlySet<unknown> = new Set(['deferred', 'requested-backend']);
134+
135+
/**
136+
* The acquisition disclosure this scenario accepts before reading depth facts off the capture,
137+
* keyed on the typed `snapshotQuality` verdict and never on the fallback warning's wording.
138+
*
139+
* A regular or raw capture of the fixture is served one of two ways, and each discloses itself
140+
* here (#3328):
141+
*
142+
* - the host AX bridge served it. The bridge publishes no quality verdict at all
143+
* (`presentIosSnapshotAcquisition` reports the tree it read without one), so an absent verdict IS
144+
* the bridge's disclosure.
145+
* - the route sent the capture to the XCTest runner instead — the bridge probe circuit is open for
146+
* this app generation, the bridge is still being prepared, a system surface is presented — and
147+
* the runner always stamps which of its own strategies served the payload. That disclosure is
148+
* legitimate on a regular snapshot: the bridge is a fast path, its circuit is per app generation,
149+
* and a runner-served capture keeps the hittability evidence this scenario asserts below.
150+
*
151+
* What the scenario may NOT be read from is a capture whose own verdict says it degraded or
152+
* served nothing: `sparse` means no backend served the screen, and `recovered` with a code other
153+
* than the two pre-selected ones means the strategy the presented depth semantics belong to failed
154+
* mid-capture and another answered (#1569 — two strategies are not comparable views of one screen).
155+
* Either one fails here with the response that proved it. A verdict whose state is outside the
156+
* kernel-declared `SNAPSHOT_QUALITY_STATES` exhausts nothing and fails too — the lane must not
157+
* certify a tree whose acquisition it does not classify.
158+
*/
159+
export function assertSimulatorSnapshotAcquisition(
160+
result: { json?: any },
161+
description: string,
162+
): void {
163+
const quality = result.json?.data?.snapshotQuality;
164+
if (quality !== undefined) {
165+
assert.ok(
166+
(SNAPSHOT_QUALITY_STATES as readonly unknown[]).includes(quality.state),
167+
`${description} disclosed a quality state outside the declared vocabulary: ${JSON.stringify(result)}`,
168+
);
169+
assert.notEqual(
170+
quality.state,
171+
'sparse',
172+
`${description} reports no backend served this screen: ${JSON.stringify(result)}`,
173+
);
174+
if (quality.state === 'recovered') {
175+
assert.ok(
176+
PRE_SELECTED_REASON_CODES.has(quality.reasonCode),
177+
`${description} fell back to another capture strategy mid-capture, so its presented depth is not comparable: ${JSON.stringify(result)}`,
178+
);
179+
}
180+
}
131181
assert.equal(
132182
result.json?.data?.warnings?.includes(MISSING_HITTABILITY_WARNING) ?? false,
133183
false,
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
import { WAIT_REASONS } from '@agent-device/contracts/wait';
2+
import type { CliJsonResult } from '../cli-json.ts';
3+
4+
/**
5+
* The wait verdicts that say the observation itself was prevented — the runner restarted,
6+
* readiness work (runner start, bridge target discovery) ate the budget, or no capture was ever
7+
* readable. Each is a lane condition on this host, not an answer about the screen: the product
8+
* itself publishes all three with `retriable: true` and a "then retry" hint on the wire
9+
* (`wait-polling.ts`), and they are the observed reds of the smoke lane (run 37356199982's
10+
* `wait_runner_restart_exhausted` with `readableCaptures: 0`).
11+
*/
12+
const OBSERVATION_PREVENTED_WAIT_REASONS: ReadonlySet<string> = new Set([
13+
WAIT_REASONS.captureStalled,
14+
WAIT_REASONS.runnerRestartExhausted,
15+
WAIT_REASONS.readinessExhausted,
16+
]);
17+
18+
/**
19+
* Whether one failed E2E step's result says the DEVICE prevented the observation, so re-issuing
20+
* the same step is a measurement of the same question rather than a second bite at a real defect.
21+
*
22+
* Keyed on the conjunction of two typed signals — the wire `retriable` verdict the daemon hoists
23+
* from its throw sites AND `error.details.reason` from the wait taxonomy — never on error text.
24+
* Either signal alone is too wide: `retriable` also rides failures the lane must not re-issue
25+
* blind (e.g. an app the lane itself is expected to have launched via another route), and a reason
26+
* set copied onto a response the product marks non-retriable would retry a verdict the product
27+
* says cannot succeed unchanged. The pairing is pinned in
28+
* `test/integration/ios-simulator-e2e-step-retry-policy.test.ts`.
29+
*
30+
* `wait_target_absent` (the screen was readable and simply did not hold the target),
31+
* `wait_deadline_exceeded` (a readable capture consumed the budget), a wrong assertion value, a
32+
* runner crash, and any non-wait failure are all non-retriable by this predicate.
33+
*/
34+
export function isObservationPreventedStepMiss(result: CliJsonResult): boolean {
35+
const error = result.json?.error;
36+
if (!error || result.status === 0) return false;
37+
return error.retriable === true && OBSERVATION_PREVENTED_WAIT_REASONS.has(error.details?.reason);
38+
}

0 commit comments

Comments
 (0)