Skip to content

Commit a35a690

Browse files
committed
ci(1874): declare the diagnose lane and read its iterations honestly
The loop that #1874 is investigated with could not tell the truth about itself. It classified every non-`passed` iteration as a stall, which after #2035 gave the looped test an XCTSkipIf meant an environment flip would report a 100% stall rate; it captured cadence only for failures, though an absorbed episode now passes; and it read its logs with shell pipelines whose exit status means "did this match", so an iteration that legitimately matched nothing killed the job before it could be summarized. scripts/diagnose-1874-iteration.ts reads one iteration: xcodebuild's own verdict, the `type-all` duration, and the cadence worth keeping. A nonzero exit outranks a green measured test — in `pair` mode the neighbour or the runner can fail while the measured test passes — and a run that produced no verdict is named as ours rather than counted as a stall. The workflow gains the #1781 lane declaration it never had, including the kill criterion that says to delete it when this closes. Closes #1874. Both filed symptoms are resolved. `smoke:form-input` was root-caused and fixed in #2035: the fixture's placeholder was identical to the value every suite filled, so `fill` could never be verified on the penalized route — deterministic, not a flake, and only visible under load because that route is gated on a penalized XCTest channel. The targeted XCTest is mitigated by the progress-aware commit budget, with 200 consecutive green loop iterations across two dispatches. The issue's remaining question — why the input pipeline throttles — is answered by the second dispatch, and the premise was wrong: it does not. Posting 17 characters took 484 ms and the commit was observed on the first poll, inside an iteration whose `type-all` measured 14334 ms. The ~12.6 s went to accessibility round-trips before any character was posted, which is #1105's path, not the input pipeline's.
1 parent 77b5be8 commit a35a690

4 files changed

Lines changed: 263 additions & 16 deletions

File tree

‎.github/workflows/1874-diagnose.yml‎

Lines changed: 37 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,27 @@
11
name: 1874 Diagnose
22

3+
# Forces the #1874 commit-stall question instead of waiting for a red lane: loops the one test
4+
# whose failures the issue records, on real CI hardware, across both arches and with/without the
5+
# neighbour test that shares the simulator.
6+
#
7+
# Under the lane rule in #1781:
8+
#
9+
# - Catches: iOS text-entry commit episodes — the simulator input pipeline delivering
10+
# synthesized characters slowly enough to matter. Since #2035 the commit wait absorbs an
11+
# episode that keeps making progress, so an episode now surfaces as a SLOW PASS rather than a
12+
# failure. This loop records the per-iteration cadence either way, which is the evidence
13+
# #1874 is still open for.
14+
# - Evidence: the 2026-08-21 dispatch passed 50/50 yet caught a sub-critical episode
15+
# (`type-all` 2487 ms across 5 polls) that no PR lane would have reported. The issue's table
16+
# also records that the episode does not reproduce locally.
17+
# - Cost: nothing while idle — `workflow_dispatch` only, and the `push:` branch does not exist
18+
# on origin. A dispatch is four macOS concurrency slots, each a runner build plus `iterations`
19+
# iterations of simulator time. It adds nothing to any PR. The `gate: swift-runner-ios` below
20+
# is the shared build action's input, not an ownership claim: ios.yml owns that gate, so
21+
# `pnpm check:gate-manifest` neither needs nor gets a declaration for this file.
22+
# - Kill criterion: delete this file in the commit that closes #1874. It answers exactly one
23+
# open question — why the pipeline throttles — and nothing else reads its artifacts.
24+
325
on:
426
workflow_dispatch:
527
inputs:
@@ -63,6 +85,8 @@ jobs:
6385
test -n "$XCTESTRUN_PATH"
6486
UDID="${{ steps.ios-simulator.outputs.simulator-udid }}"
6587
ITER="${{ github.event.inputs.iterations || '25' }}"
88+
TEST_NAME=testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden
89+
mkdir -p .tmp
6690
6791
if [ "${{ matrix.mode }}" = "pair" ]; then
6892
EXTRA_ONLY_TESTING=(-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBareDelayedTypeFailsWhenTappedInputDisappearsMidCommand)
@@ -73,33 +97,30 @@ jobs:
7397
sys_prof="$(system_profiler SPHardwareDataType 2>/dev/null | grep -E 'Chip|Cores|Memory' || true)"
7498
echo "HOST: $sys_prof" | tee stall-summary.txt
7599
76-
pass=0; fail=0
100+
pass=0; skipped=0; failed=0; noresult=0; runfailed=0
77101
for i in $(seq 1 "$ITER"); do
78102
LOG=".tmp/stall-run-$i.log"
79103
set +e
80104
xcodebuild test-without-building \
81105
-xctestrun "$XCTESTRUN_PATH" \
82106
-destination "platform=iOS Simulator,id=$UDID" \
83-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden \
107+
-only-testing:"AgentDeviceRunnerUITests/RunnerTests/$TEST_NAME" \
84108
${EXTRA_ONLY_TESTING[@]+"${EXTRA_ONLY_TESTING[@]}"} \
85109
> "$LOG" 2>&1
86110
rc=$?
87111
set -e
88-
if grep -q "testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden]' passed" "$LOG"; then
89-
pass=$((pass+1)); verdict=pass
90-
else
91-
fail=$((fail+1)); verdict=STALL
92-
fi
93-
phase="$(grep -o 'phase=type-all durationMs=[0-9.]*' "$LOG" | tail -1)"
94-
polls="$(grep -c 'DEBUG-1874] poll' "$LOG" || true)"
95-
echo "iter=$i verdict=$verdict rc=$rc $phase polls=$polls" | tee -a stall-summary.txt
96-
if [ "$verdict" = "STALL" ]; then
97-
grep 'DEBUG-1874' "$LOG" | head -40 | tee -a stall-summary.txt
98-
cp "$LOG" ".tmp/stall-failure-$i.log"
99-
fi
112+
verdict="$(node --experimental-strip-types scripts/diagnose-1874-iteration.ts \
113+
"$LOG" "$TEST_NAME" "$i" "$rc")"
114+
case "$verdict" in
115+
passed) pass=$((pass+1)) ;;
116+
skipped) skipped=$((skipped+1)) ;;
117+
no-result) noresult=$((noresult+1)) ;;
118+
run-failed) runfailed=$((runfailed+1)) ;;
119+
*) failed=$((failed+1)) ;;
120+
esac
100121
done
101122
102-
echo "RESULT arch=${{ matrix.arch }} mode=${{ matrix.mode }}: $pass passed, $fail stalled of $ITER" | tee -a stall-summary.txt
123+
echo "RESULT arch=${{ matrix.arch }} mode=${{ matrix.mode }}: $pass passed, $skipped skipped, $failed failed, $noresult no-result, $runfailed run-failed of $ITER" | tee -a stall-summary.txt
103124
104125
- name: Upload stall evidence
105126
if: always()
@@ -109,5 +130,5 @@ jobs:
109130
include-hidden-files: true
110131
path: |
111132
stall-summary.txt
112-
.tmp/stall-failure-*.log
133+
.tmp/stall-evidence-*.log
113134
if-no-files-found: ignore
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import { execFileSync } from 'node:child_process';
2+
import fs from 'node:fs';
3+
import os from 'node:os';
4+
import path from 'node:path';
5+
import { fileURLToPath } from 'node:url';
6+
import { expect, test } from 'vitest';
7+
import { readIteration, type IterationReport } from '../diagnose-1874-iteration.ts';
8+
9+
const script = path.join(
10+
path.dirname(fileURLToPath(import.meta.url)),
11+
'../diagnose-1874-iteration.ts',
12+
);
13+
const NAME = 'testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden';
14+
const POLL = '[DEBUG-1874] poll t=1ms observedLen=0 expectedPrefixLen=0';
15+
const CADENCE = [
16+
'[DEBUG-1874] wait start expectedLen=17',
17+
POLL,
18+
'[DEBUG-1874] wait outcome=settled',
19+
];
20+
21+
const verdictLine = (test: string, word: string) =>
22+
`Test Case '-[X.RunnerTests ${test}]' ${word} (1.0 seconds).`;
23+
const phases = (typeAllMs: string) =>
24+
[
25+
`phase=focus durationMs=644.6`,
26+
`phase=type-all durationMs=${typeAllMs}`,
27+
`phase=total durationMs=7379.7`,
28+
].map((phase) => `AGENT_DEVICE_RUNNER_TEXT_ENTRY_PHASE commandId=c ${phase} chars=17`);
29+
30+
const SHAPES: readonly (Partial<IterationReport> & { name: string; log: string[]; rc?: number })[] =
31+
[
32+
{
33+
name: 'healthy pass',
34+
log: [verdictLine(NAME, 'passed'), ...phases('796.1'), ...CADENCE],
35+
verdict: 'passed',
36+
keepEvidence: false,
37+
lines: ['iter=1 verdict=passed rc=0 durationMs=796 polls=1'],
38+
},
39+
{
40+
name: 'slow pass — an absorbed episode, which is the evidence #1874 wants',
41+
log: [verdictLine(NAME, 'passed'), ...phases('14146.4'), ...CADENCE],
42+
verdict: 'passed',
43+
keepEvidence: true,
44+
lines: ['iter=1 verdict=passed rc=0 durationMs=14146 polls=1', ...CADENCE],
45+
},
46+
{
47+
name: 'skipped — an environment flip, not a stall',
48+
log: [verdictLine(NAME, 'skipped')],
49+
verdict: 'skipped',
50+
keepEvidence: true,
51+
lines: ['iter=1 verdict=skipped rc=0 durationMs=0 polls=0'],
52+
},
53+
{
54+
name: 'no verdict at all — ours to own, not the product’s',
55+
log: ['Executed 0 tests, with 0 failures (0 unexpected) in 0.000 seconds'],
56+
verdict: 'no-result',
57+
keepEvidence: true,
58+
},
59+
{
60+
name: 'a verdict word this loop does not model is not a stall either',
61+
log: [verdictLine(NAME, 'errored')],
62+
verdict: 'no-result',
63+
},
64+
{
65+
name: 'pair mode — the neighbour test logs first',
66+
log: [
67+
verdictLine('testBareDelayedTypeFailsWhenTappedInputDisappearsMidCommand', 'passed'),
68+
'AGENT_DEVICE_RUNNER_TEXT_ENTRY_PHASE commandId=n phase=type-all durationMs=5000.0 chars=17',
69+
verdictLine(NAME, 'passed'),
70+
...phases('812.0'),
71+
],
72+
verdict: 'passed',
73+
keepEvidence: false,
74+
},
75+
{
76+
name: 'restarted — the final word is the outcome',
77+
log: [verdictLine(NAME, 'failed'), ...phases('900.0'), verdictLine(NAME, 'passed')],
78+
verdict: 'passed',
79+
keepEvidence: false,
80+
},
81+
{
82+
name: 'measured test passed, xcodebuild did not',
83+
log: [verdictLine(NAME, 'passed'), ...phases('796.1'), ...CADENCE],
84+
rc: 65,
85+
verdict: 'run-failed',
86+
keepEvidence: true,
87+
},
88+
{
89+
name: 'louder than the cadence cap',
90+
log: [verdictLine(NAME, 'failed'), ...Array.from({ length: 45 }, () => POLL)],
91+
verdict: 'failed',
92+
keepEvidence: true,
93+
lines: [
94+
'iter=1 verdict=failed rc=0 durationMs=0 polls=45',
95+
...Array.from({ length: 40 }, () => POLL),
96+
'… 5 more DEBUG-1874 lines (full log in the artifact)',
97+
],
98+
},
99+
];
100+
101+
test.each(SHAPES)('reads a $name iteration', (shape) => {
102+
const report = readIteration(shape.log.join('\n'), NAME, 1, shape.rc ?? 0);
103+
expect(report.verdict).toBe(shape.verdict);
104+
if (shape.keepEvidence !== undefined) expect(report.keepEvidence).toBe(shape.keepEvidence);
105+
if (shape.lines) expect(report.lines).toEqual(shape.lines);
106+
});
107+
108+
test('the test name is matched literally, not as a pattern', () => {
109+
const log = verdictLine('testAXB', 'passed');
110+
expect(readIteration(log, 'testA.B', 1, 0).verdict).toBe('no-result');
111+
expect(readIteration(log, 'testAXB', 1, 0).verdict).toBe('passed');
112+
});
113+
114+
test.each([
115+
{ label: 'slow pass', rc: 0, verdict: 'passed' },
116+
{ label: 'nonzero exit over a green measured test', rc: 65, verdict: 'run-failed' },
117+
])(
118+
'the entry point writes, keeps evidence and prints the verdict for a $label',
119+
({ rc, verdict }) => {
120+
const cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'diagnose-1874-'));
121+
try {
122+
fs.mkdirSync(path.join(cwd, '.tmp'));
123+
const log = path.join(cwd, 'iteration.log');
124+
fs.writeFileSync(
125+
log,
126+
[verdictLine(NAME, 'passed'), ...phases('14146.4'), ...CADENCE].join('\n'),
127+
);
128+
129+
const printed = execFileSync(
130+
process.execPath,
131+
['--experimental-strip-types', script, log, NAME, '7', `${rc}`],
132+
{ cwd, encoding: 'utf8' },
133+
);
134+
135+
expect(printed).toBe(verdict);
136+
expect(fs.readFileSync(path.join(cwd, 'stall-summary.txt'), 'utf8')).toBe(
137+
[`iter=7 verdict=${verdict} rc=${rc} durationMs=14146 polls=1`, ...CADENCE, ''].join('\n'),
138+
);
139+
expect(fs.existsSync(path.join(cwd, '.tmp', 'stall-evidence-7.log'))).toBe(true);
140+
} finally {
141+
fs.rmSync(cwd, { recursive: true, force: true });
142+
}
143+
},
144+
);

‎scripts/diagnose-1874-iteration.ts‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
// Reads one iteration of the #1874 diagnose loop (.github/workflows/1874-diagnose.yml).
2+
3+
import fs from 'node:fs';
4+
import path from 'node:path';
5+
import { pathToFileURL } from 'node:url';
6+
7+
/** Measured on CI: healthy iterations 628-1131 ms (n=50), the episode on record 2487 ms. */
8+
const SLOW_PASS_MS = 2000;
9+
const CADENCE_LIMIT = 40;
10+
const SUMMARY = 'stall-summary.txt';
11+
const EVIDENCE_DIR = '.tmp';
12+
13+
type MeasuredVerdict = 'passed' | 'skipped' | 'failed';
14+
15+
export type IterationReport = {
16+
readonly verdict: MeasuredVerdict | 'no-result' | 'run-failed';
17+
readonly keepEvidence: boolean;
18+
readonly lines: readonly string[];
19+
};
20+
21+
export function readIteration(
22+
log: string,
23+
testName: string,
24+
iteration: number,
25+
rc: number,
26+
): IterationReport {
27+
const lines = log.split('\n');
28+
const marker = `${testName}]' `;
29+
const word = lines
30+
.filter((line) => line.includes(marker))
31+
.map((line) => line.slice(line.indexOf(marker) + marker.length).split(' ')[0])
32+
.at(-1);
33+
const measured = isMeasuredVerdict(word) ? word : 'no-result';
34+
const exitContradictsMeasured = rc !== 0 && measured !== 'failed' && measured !== 'no-result';
35+
const verdict = exitContradictsMeasured ? 'run-failed' : measured;
36+
const durationMs = Number(
37+
lines.flatMap((line) => /phase=type-all durationMs=(\d+)/.exec(line)?.[1] ?? []).at(-1) ?? 0,
38+
);
39+
const cadence = lines.filter((line) => line.includes('[DEBUG-1874]'));
40+
const keepEvidence = verdict !== 'passed' || durationMs > SLOW_PASS_MS;
41+
42+
const polls = cadence.filter((line) => line.includes('] poll')).length;
43+
const summary = `iter=${iteration} verdict=${verdict} rc=${rc} durationMs=${durationMs} polls=${polls}`;
44+
const dropped = cadence.length - CADENCE_LIMIT;
45+
return {
46+
verdict,
47+
keepEvidence,
48+
lines: !keepEvidence
49+
? [summary]
50+
: [
51+
summary,
52+
...cadence.slice(0, CADENCE_LIMIT),
53+
...(dropped > 0 ? [`… ${dropped} more DEBUG-1874 lines (full log in the artifact)`] : []),
54+
],
55+
};
56+
}
57+
58+
function isMeasuredVerdict(word: string | undefined): word is MeasuredVerdict {
59+
return word === 'passed' || word === 'skipped' || word === 'failed';
60+
}
61+
62+
function main(): number {
63+
const [logPath, testName, iteration, rc] = process.argv.slice(2);
64+
if (!logPath || !testName || !iteration || !rc) {
65+
throw new Error('usage: diagnose-1874-iteration.ts <log> <testName> <iteration> <rc>');
66+
}
67+
const report = readIteration(
68+
fs.readFileSync(logPath, 'utf8'),
69+
testName,
70+
Number(iteration),
71+
Number(rc),
72+
);
73+
fs.appendFileSync(SUMMARY, `${report.lines.join('\n')}\n`);
74+
if (report.keepEvidence) {
75+
fs.copyFileSync(logPath, path.join(EVIDENCE_DIR, `stall-evidence-${iteration}.log`));
76+
}
77+
process.stdout.write(report.verdict);
78+
return 0;
79+
}
80+
81+
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) process.exit(main());

‎vitest.config.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,7 @@ export default defineConfig({
153153
// Parses CI configuration only, so this action guard needs no device or subprocess lane.
154154
'test/ci/upload-agent-device-artifacts.test.ts',
155155
'test/ci/upload-artifact-hidden-paths.test.ts',
156+
'scripts/__tests__/diagnose-1874-iteration.test.ts',
156157
// The size reporter is preserved across a base checkout; its entrypoint and imported
157158
// modules must move as one directory or the Bundle Size lane fails before measuring.
158159
'test/ci/size-workflow.test.ts',

0 commit comments

Comments
 (0)