Skip to content

Commit 5345d94

Browse files
committed
ci(1874): declare the diagnose lane and stop it misreading its own loop
Two defects in the harness #1874 depends on, both found while dispatching it after #2035 merged. It classified every non-`passed` iteration as a stall. #2035 gave the looped test an XCTSkipIf on an ambient software keyboard, so an environment flip would now be reported as a 100% stall rate; a selection matching nothing reported the same. It reads xcodebuild's own verdict word instead, counting passed/skipped/ failed/no-result separately. It captured cadence only for failing iterations. Since #2035 the commit wait absorbs an episode that keeps making progress, so an episode passes — the loop was reporting only the episodes bad enough to still be red. It now also captures any iteration slower than SLOW_PASS_MS. That threshold is measured, not guessed: the 2026-08-26 dispatch ran 50 healthy iterations at 628-1131 ms across two arm64 legs, against a 2487 ms/5-poll episode on record and two single-poll outliers of 4.0 s and 14.1 s in that same dispatch. Keyed on elapsed time rather than poll count because those outliers settled on the FIRST poll: the time went somewhere before the wait, which is precisely the evidence the dumps were not collecting. Also adds the #1781 lane declaration the file never had — Catches, Evidence, Cost, and a Kill criterion that names the commit which should delete it. No gate manifest entry: the `gate: swift-runner-ios` input here is the shared build action's, and ios.yml owns that gate.
1 parent ea1d6b8 commit 5345d94

1 file changed

Lines changed: 52 additions & 11 deletions

File tree

‎.github/workflows/1874-diagnose.yml‎

Lines changed: 52 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,27 @@
11
name: 1874 Diagnose
22

3+
# Forces the #1874 commit-stall question instead of waiting for a red lane: loops the one test
4+
# whose failures the issue records, on real CI hardware, across both arches and with/without the
5+
# neighbour test that shares the simulator.
6+
#
7+
# Under the lane rule in #1781:
8+
#
9+
# - Catches: iOS text-entry commit episodes — the simulator input pipeline delivering
10+
# synthesized characters slowly enough to matter. Since #2035 the commit wait absorbs an
11+
# episode that keeps making progress, so an episode now surfaces as a SLOW PASS rather than a
12+
# failure. This loop records the per-iteration cadence either way, which is the evidence
13+
# #1874 is still open for.
14+
# - Evidence: the 2026-08-21 dispatch passed 50/50 yet caught a sub-critical episode
15+
# (`type-all` 2487 ms across 5 polls) that no PR lane would have reported. The issue's table
16+
# also records that the episode does not reproduce locally.
17+
# - Cost: nothing while idle — `workflow_dispatch` only, and the `push:` branch does not exist
18+
# on origin. A dispatch is four macOS concurrency slots, each a runner build plus `iterations`
19+
# iterations of simulator time. It adds nothing to any PR. The `gate: swift-runner-ios` below
20+
# is the shared build action's input, not an ownership claim: ios.yml owns that gate, so
21+
# `pnpm check:gate-manifest` neither needs nor gets a declaration for this file.
22+
# - Kill criterion: delete this file in the commit that closes #1874. It answers exactly one
23+
# open question — why the pipeline throttles — and nothing else reads its artifacts.
24+
325
on:
426
workflow_dispatch:
527
inputs:
@@ -63,6 +85,16 @@ jobs:
6385
test -n "$XCTESTRUN_PATH"
6486
UDID="${{ steps.ios-simulator.outputs.simulator-udid }}"
6587
ITER="${{ github.event.inputs.iterations || '25' }}"
88+
TEST_NAME=testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden
89+
# Above this, capture the cadence even though the iteration passed. Keyed on elapsed
90+
# time, not poll count: the 2026-08-26 dispatch of this workflow found both shapes, and
91+
# only one of them is poll-heavy. Its healthy iterations ran 628-1131 ms (n=50, median
92+
# 796/702 ms per leg), against a 2487 ms/5-poll episode on record and two single-poll
93+
# outliers of 4.0 s and 14.1 s — where the commit settled on the FIRST poll, so the time
94+
# was spent before the wait, not in it. 2000 ms sits above every healthy iteration
95+
# measured and below every episode.
96+
SLOW_PASS_MS=2000
97+
mkdir -p .tmp
6698
6799
if [ "${{ matrix.mode }}" = "pair" ]; then
68100
EXTRA_ONLY_TESTING=(-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBareDelayedTypeFailsWhenTappedInputDisappearsMidCommand)
@@ -73,33 +105,42 @@ jobs:
73105
sys_prof="$(system_profiler SPHardwareDataType 2>/dev/null | grep -E 'Chip|Cores|Memory' || true)"
74106
echo "HOST: $sys_prof" | tee stall-summary.txt
75107
76-
pass=0; fail=0
108+
pass=0; skipped=0; fail=0
77109
for i in $(seq 1 "$ITER"); do
78110
LOG=".tmp/stall-run-$i.log"
79111
set +e
80112
xcodebuild test-without-building \
81113
-xctestrun "$XCTESTRUN_PATH" \
82114
-destination "platform=iOS Simulator,id=$UDID" \
83-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden \
115+
-only-testing:"AgentDeviceRunnerUITests/RunnerTests/$TEST_NAME" \
84116
${EXTRA_ONLY_TESTING[@]+"${EXTRA_ONLY_TESTING[@]}"} \
85117
> "$LOG" 2>&1
86118
rc=$?
87119
set -e
88-
if grep -q "testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden]' passed" "$LOG"; then
89-
pass=$((pass+1)); verdict=pass
90-
else
91-
fail=$((fail+1)); verdict=STALL
92-
fi
120+
# xcodebuild's own verdict word, not pass-or-everything-else: since #2035 the test
121+
# can legitimately skip (XCTSkipIf on an ambient software keyboard), and a selection
122+
# that matches nothing reports no verdict at all. Counting either as a stall would
123+
# manufacture a 100% failure rate out of an environment flip or a bad selector.
124+
verdict="$(sed -n "s/.*$TEST_NAME]' \([a-z]*\).*/\1/p" "$LOG" | tail -1)"
125+
verdict="${verdict:-no-result}"
93126
phase="$(grep -o 'phase=type-all durationMs=[0-9.]*' "$LOG" | tail -1)"
127+
duration_ms="${phase##*=}"; duration_ms="${duration_ms%%.*}"; duration_ms="${duration_ms:-0}"
94128
polls="$(grep -c 'DEBUG-1874] poll' "$LOG" || true)"
129+
case "$verdict" in
130+
passed) pass=$((pass+1)) ;;
131+
skipped) skipped=$((skipped+1)) ;;
132+
*) fail=$((fail+1)) ;;
133+
esac
95134
echo "iter=$i verdict=$verdict rc=$rc $phase polls=$polls" | tee -a stall-summary.txt
96-
if [ "$verdict" = "STALL" ]; then
135+
# The cadence is the artifact. An absorbed episode passes, so keying the dump on
136+
# failure alone would report only the episodes bad enough to still be red.
137+
if [ "$verdict" != passed ] || [ "$duration_ms" -gt "$SLOW_PASS_MS" ]; then
97138
grep 'DEBUG-1874' "$LOG" | head -40 | tee -a stall-summary.txt
98-
cp "$LOG" ".tmp/stall-failure-$i.log"
139+
cp "$LOG" ".tmp/stall-evidence-$i.log"
99140
fi
100141
done
101142
102-
echo "RESULT arch=${{ matrix.arch }} mode=${{ matrix.mode }}: $pass passed, $fail stalled of $ITER" | tee -a stall-summary.txt
143+
echo "RESULT arch=${{ matrix.arch }} mode=${{ matrix.mode }}: $pass passed, $skipped skipped, $fail stalled of $ITER" | tee -a stall-summary.txt
103144
104145
- name: Upload stall evidence
105146
if: always()
@@ -108,5 +149,5 @@ jobs:
108149
name: stall-evidence-${{ matrix.arch }}-${{ matrix.mode }}
109150
path: |
110151
stall-summary.txt
111-
.tmp/stall-failure-*.log
152+
.tmp/stall-evidence-*.log
112153
if-no-files-found: ignore

0 commit comments

Comments
 (0)