Skip to content

Commit 80075b6

Browse files
authored
ci(conformance): keep the Maestro version probe's failure evidence (#4503)
* ci(conformance): keep the Maestro version probe's failure evidence (#4455) The layer-3 pin check parsed `maestro --version` inside a pipeline that sent stderr to /dev/null and masked the pipeline exit with `|| true`. Its whole failure record was `expected=<pin> actual=`, so the November 9 run could not tell a CLI that never started from one that printed nothing parseable. Run the probe through scripts/maestro-conformance/verify-cli-version.mjs. It records the exit status, signal, spawn error and both streams, retains each stream's last 8 KiB newest-first, and reports one verdict: command failure, missing or unparseable version, mismatch, or match. The pin invariant is unchanged and still fails closed; a probe that did not complete cleanly is never accepted on the strength of its stdout. The fixture-executable suite covers a matching version inside notices, a wrong version, empty and unparseable stdout, a nonzero exit with matching-looking stdout, an absent executable, and bounded oversized output. * fix(conformance): parse probe versions before diagnostic truncation
1 parent ed73e79 commit 80075b6

5 files changed

Lines changed: 499 additions & 14 deletions

File tree

‎.github/workflows/conformance-differential.yml‎

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -117,19 +117,13 @@ jobs:
117117
curl -fsSL "https://get.maestro.mobile.dev" | bash
118118
echo "$HOME/.maestro/bin" >> "$GITHUB_PATH"
119119
120+
# Asserts what `maestro --version` actually reports against the same pin layers 1-2 use.
121+
# The probe lives in scripts/maestro-conformance so its failure keeps the command's exit,
122+
# stdout and stderr as bounded diagnostics and names which of command failure,
123+
# missing/unparseable version, or a genuine mismatch it saw (#4455). It fails closed: only a
124+
# reported version equal to the pin gets the differential to run.
120125
- name: Verify the Maestro CLI matches the oracle pin
121-
run: |
122-
set -euo pipefail
123-
EXPECTED=$(node -p "require('./scripts/maestro-conformance/pinned-upstream.json').version")
124-
# `maestro --version` can print notices (e.g. the analytics banner)
125-
# around the version, so match the semver line rather than the whole
126-
# output. Empty ACTUAL fails the comparison below, as it should.
127-
ACTUAL=$(maestro --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | tail -1 || true)
128-
echo "expected=$EXPECTED actual=$ACTUAL"
129-
if [ "$ACTUAL" != "$EXPECTED" ]; then
130-
echo "::error::Layer-3 Maestro CLI is $ACTUAL but the oracle pins $EXPECTED. A differential against a different Maestro proves nothing about the pinned behavior."
131-
exit 1
132-
fi
126+
run: node scripts/maestro-conformance/verify-cli-version.mjs
133127

134128
# --trace-root lets the runner find agent-device's replay-timing.ndjson and
135129
# assert engine-side invariants (e.g. the settle loop latches instead of

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@
125125
"ad": "node bin/agent-device.mjs",
126126
"bench:android:automation": "node scripts/bench-android-automation.mjs",
127127
"bench:help-conformance": "node scripts/help-conformance-bench.mjs",
128-
"maestro:conformance": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/maestro-conformance/format-generated-json.test.mjs packages/maestro/test/conformance/verify.test.ts packages/maestro/test/conformance/differential/engine-process.test.ts packages/maestro/test/conformance/differential/report-output.test.ts packages/maestro/test/conformance/differential/run.test.ts packages/maestro/test/conformance/differential/invariants.test.ts",
128+
"maestro:conformance": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/maestro-conformance/format-generated-json.test.mjs scripts/maestro-conformance/verify-cli-version.test.mjs packages/maestro/test/conformance/verify.test.ts packages/maestro/test/conformance/differential/engine-process.test.ts packages/maestro/test/conformance/differential/report-output.test.ts packages/maestro/test/conformance/differential/run.test.ts packages/maestro/test/conformance/differential/invariants.test.ts",
129129
"maestro:conformance:regenerate": "node --experimental-strip-types scripts/maestro-conformance/regenerate.mjs",
130130
"maestro:conformance:differential": "node --experimental-strip-types packages/maestro/test/conformance/differential/run.ts",
131131
"test:ios-snapshot-differential": "node --experimental-strip-types scripts/swift-toolchain-tmpdir.ts swift test --package-path apple/snapshot-presentation && node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/ios-snapshot-differential.test.ts",

‎scripts/maestro-conformance/README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,11 @@ Runs scheduled on the `conformance-differential` workflow, and on demand via
147147

148148
The workflow builds and installs the fixture app, verifies its bundle id, pins
149149
the Maestro CLI to the same version as layers 1-2, and passes `--maestro` so the
150-
flow routes through the compat engine.
150+
flow routes through the compat engine. The pin check runs
151+
[`verify-cli-version.mjs`](./verify-cli-version.mjs): it fails unless
152+
`maestro --version` exits 0 and its last semver in stdout equals the pin, and it keeps the
153+
probe's exit, stdout and stderr as bounded diagnostics so a failure names its cause
154+
(#4455).
151155

152156
The built `.app` is **cached** (keyed on the app's sources, its dependency graph,
153157
the iOS runtime, and the Xcode version), because building it costs ~22 minutes
Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
#!/usr/bin/env node
2+
// Asserts that the installed Maestro CLI reports the version this oracle pins, and keeps the
3+
// probe's exit status, stdout and stderr as the failure evidence (#4455).
4+
//
5+
// The workflow step this replaced parsed the version inside a shell pipeline that sent the CLI's
6+
// stderr to /dev/null and masked the pipeline's exit with `|| true`. Its whole failure record was
7+
// `expected=<pin> actual=`, which cannot tell apart the causes that need different fixes: the CLI
8+
// never started, it started and printed nothing parseable, or a genuinely different Maestro is
9+
// installed. The invariant stays strict — anything other than the pinned version fails the step —
10+
// because a differential against an unproven CLI proves nothing about pinned behavior. What was
11+
// missing is the cause, which is what this reports.
12+
//
13+
// A script rather than `run:` shell because retained evidence has to be bounded: a JVM startup
14+
// failure prints a stack trace, and a step log is not the place to dump one unbounded. Each stream
15+
// keeps its last MAX_DIAGNOSTIC_BYTES. Version matching consumes the full stream separately, so
16+
// truncating diagnostics cannot change which version is checked.
17+
import { spawn } from 'node:child_process';
18+
import fs from 'node:fs';
19+
import path from 'node:path';
20+
import { fileURLToPath, pathToFileURL } from 'node:url';
21+
22+
const HERE = path.dirname(fileURLToPath(import.meta.url));
23+
24+
/** Per stream: how much of the probe's output the run retains and prints. */
25+
export const MAX_DIAGNOSTIC_BYTES = 8192;
26+
27+
/** Every verdict the probe can reach. Only `match` lets the differential proceed. */
28+
const PROBE_OUTCOMES = Object.freeze({
29+
match: 'match',
30+
commandFailed: 'command-failed',
31+
missingVersion: 'missing-version',
32+
unparseableVersion: 'unparseable-version',
33+
versionMismatch: 'version-mismatch',
34+
});
35+
36+
/** Reads the pin layers 1-2 use, so the two cannot drift apart. */
37+
export function readPinnedVersion() {
38+
const { version } = JSON.parse(fs.readFileSync(path.join(HERE, 'pinned-upstream.json'), 'utf8'));
39+
if (typeof version !== 'string' || version === '') {
40+
throw new Error(`pinned-upstream.json declares no version to check the Maestro CLI against`);
41+
}
42+
return version;
43+
}
44+
45+
/** The last semver in `output`, with oversized versions represented by a bounded failure marker. */
46+
export function parseVersion(output) {
47+
const capture = createVersionCapture();
48+
capture.append(output);
49+
return capture.read();
50+
}
51+
52+
const VERSION_STATES = {
53+
search: { digit: 'major' },
54+
major: { digit: 'major', dot: 'major-dot' },
55+
'major-dot': { digit: 'minor' },
56+
minor: { digit: 'minor', dot: 'minor-dot' },
57+
'minor-dot': { digit: 'patch' },
58+
patch: { digit: 'patch' },
59+
};
60+
61+
function versionCharacterKind(byte) {
62+
if (byte >= 48 && byte <= 57) return 'digit';
63+
if (byte === 46) return 'dot';
64+
return 'other';
65+
}
66+
67+
function createVersionCapture() {
68+
let state = 'search';
69+
let candidate = '';
70+
let oversized = false;
71+
let lastVersion = null;
72+
const currentVersion = () =>
73+
oversized ? `(version exceeds ${MAX_DIAGNOSTIC_BYTES} bytes)` : candidate;
74+
const retain = (character) => {
75+
if (candidate.length < MAX_DIAGNOSTIC_BYTES) candidate += character;
76+
else oversized = true;
77+
};
78+
return {
79+
append(output) {
80+
for (const byte of Buffer.isBuffer(output) ? output : Buffer.from(output)) {
81+
const nextState = VERSION_STATES[state][versionCharacterKind(byte)];
82+
if (nextState !== undefined) {
83+
state = nextState;
84+
retain(String.fromCharCode(byte));
85+
continue;
86+
}
87+
if (state === 'patch') lastVersion = currentVersion();
88+
state = 'search';
89+
candidate = '';
90+
oversized = false;
91+
}
92+
},
93+
read() {
94+
return state === 'patch' ? currentVersion() : lastVersion;
95+
},
96+
};
97+
}
98+
99+
/**
100+
* Runs the version probe and records what it did rather than what a pipeline survived: exit status,
101+
* terminating signal, a spawn failure's code, and both output streams.
102+
*/
103+
function probeVersion({ command, args = ['--version'], env = process.env }) {
104+
return new Promise((resolve) => {
105+
const child = spawn(command, args, { env, stdio: ['ignore', 'pipe', 'pipe'] });
106+
const stdout = boundedCapture();
107+
const stderr = boundedCapture();
108+
const version = createVersionCapture();
109+
child.stdout.on('data', (chunk) => {
110+
stdout.append(chunk);
111+
version.append(chunk);
112+
});
113+
child.stderr.on('data', (chunk) => stderr.append(chunk));
114+
115+
let settled = false;
116+
child.on('close', (exitCode, signal) => {
117+
if (settled) return;
118+
settled = true;
119+
resolve({
120+
command,
121+
args,
122+
spawnErrorCode: undefined,
123+
exitCode,
124+
signal,
125+
version: version.read(),
126+
stdout: stdout.read(),
127+
stderr: stderr.read(),
128+
});
129+
});
130+
// A failed spawn (an absent or non-executable `maestro`) reports here and never closes.
131+
child.on('error', (error) => {
132+
if (settled) return;
133+
settled = true;
134+
resolve({
135+
command,
136+
args,
137+
spawnErrorCode: error?.code,
138+
exitCode: null,
139+
signal: null,
140+
version: version.read(),
141+
stdout: stdout.read(),
142+
stderr: stderr.read(),
143+
});
144+
});
145+
});
146+
}
147+
148+
/**
149+
* Decides the step's verdict. A probe that did not complete cleanly is never accepted on the
150+
* strength of its stdout: a CLI that prints the pinned version and still exits nonzero is a CLI
151+
* whose behavior the differential cannot trust.
152+
*/
153+
export function evaluateVersionProbe(probe, expected) {
154+
const version = probe.version;
155+
const outcome = probeOutcome(probe, version, expected);
156+
return { outcome, version, report: reportFor(probe, { outcome, version, expected }) };
157+
}
158+
159+
function probeOutcome(probe, version, expected) {
160+
if (probe.spawnErrorCode !== undefined || probe.signal !== null || probe.exitCode !== 0) {
161+
return PROBE_OUTCOMES.commandFailed;
162+
}
163+
if (version !== null) {
164+
return version === expected ? PROBE_OUTCOMES.match : PROBE_OUTCOMES.versionMismatch;
165+
}
166+
return probe.stdout.value === ''
167+
? PROBE_OUTCOMES.missingVersion
168+
: PROBE_OUTCOMES.unparseableVersion;
169+
}
170+
171+
const OUTCOME_EXPLANATION = {
172+
[PROBE_OUTCOMES.commandFailed]: 'did not complete successfully',
173+
[PROBE_OUTCOMES.missingVersion]: 'printed no stdout at all',
174+
[PROBE_OUTCOMES.unparseableVersion]: 'printed no version-shaped text',
175+
[PROBE_OUTCOMES.versionMismatch]: 'reports a different Maestro version',
176+
[PROBE_OUTCOMES.match]: 'reports the pinned version',
177+
};
178+
179+
function reportFor(probe, { outcome, version, expected }) {
180+
const facts = [
181+
`expected=${expected} actual=${version ?? ''}`,
182+
`probe=${probe.command} ${probe.args.join(' ')}` +
183+
` exit=${probe.exitCode ?? '-'} signal=${probe.signal ?? '-'}` +
184+
` spawnError=${probe.spawnErrorCode ?? '-'}`,
185+
`verdict=${outcome} (${OUTCOME_EXPLANATION[outcome]})`,
186+
];
187+
if (outcome === PROBE_OUTCOMES.match) {
188+
return { stdout: facts, stderr: [] };
189+
}
190+
const error =
191+
`::error::Layer-3 Maestro CLI check failed with verdict ${outcome}: the installed ` +
192+
`maestro ${OUTCOME_EXPLANATION[outcome]}. The oracle pins ${expected}; a differential ` +
193+
'against a different or unhealthy Maestro proves nothing about the pinned behavior.';
194+
return {
195+
stdout: [...facts, error, describeStream(probe.stdout, 'stdout')],
196+
stderr: [describeStream(probe.stderr, 'stderr')],
197+
};
198+
}
199+
200+
function describeStream(stream, name) {
201+
const bound =
202+
`(retained the last ${stream.bytes} of ${stream.totalBytes} bytes` +
203+
(stream.droppedBytes > 0 ? `, dropped ${stream.droppedBytes}` : '') +
204+
')';
205+
return [`--- maestro --version ${name} ${bound} ---`, stream.value.trim() || '(empty)'].join(
206+
'\n',
207+
);
208+
}
209+
210+
/** Retains at most `MAX_DIAGNOSTIC_BYTES` of a stream, newest first, and counts what it dropped. */
211+
function boundedCapture() {
212+
let totalBytes = 0;
213+
let retainedBytes = 0;
214+
const chunks = [];
215+
return {
216+
append(chunk) {
217+
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(String(chunk));
218+
totalBytes += buffer.length;
219+
retainedBytes += buffer.length;
220+
chunks.push(buffer);
221+
// One failing JVM can report its whole stack in a single write, so the newest chunk is
222+
// trimmed too: bounding only whole chunks would keep whatever one write carried.
223+
while (retainedBytes > MAX_DIAGNOSTIC_BYTES) {
224+
const oldest = chunks[0];
225+
const excess = retainedBytes - MAX_DIAGNOSTIC_BYTES;
226+
if (oldest.length <= excess) {
227+
chunks.shift();
228+
retainedBytes -= oldest.length;
229+
} else {
230+
chunks[0] = oldest.subarray(excess);
231+
retainedBytes -= excess;
232+
}
233+
}
234+
},
235+
read() {
236+
const value = Buffer.concat(chunks).toString('utf8');
237+
return { value, bytes: retainedBytes, totalBytes, droppedBytes: totalBytes - retainedBytes };
238+
},
239+
};
240+
}
241+
242+
async function main({ command = 'maestro', env = process.env } = {}) {
243+
const expected = readPinnedVersion();
244+
const { outcome, report } = evaluateVersionProbe(await probeVersion({ command, env }), expected);
245+
for (const line of report.stdout) process.stdout.write(`${line}\n`);
246+
for (const line of report.stderr) process.stderr.write(`${line}\n`);
247+
return outcome === PROBE_OUTCOMES.match ? 0 : 1;
248+
}
249+
250+
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
251+
process.exitCode = await main();
252+
}

0 commit comments

Comments
 (0)