Repository navigation
Description
Symptom
The iOS smoke lane fails on cold macOS runners with a toolchain probe timeout, unrelated to the PR under test:
Verify clean-installed Simulator snapshot bridge preparation:AppError: xcrun timed out after 10000ms(fromverify-installed-snapshot-bridge.ts→ensureSnapshotBridgeBinary).Preflight iOS runner through public CLI:Could not read the Xcode toolchain versions the Apple runner cache is keyed on (xcodebuild -version: … timed out after 5000ms).
Occurrences in the last three days, all on the same step and with no code in the failing path changed by the branch: refactor/core-replay-divergence-* (2026-09-07 16:06), claude/2339-contracts-single-owner-moves (09-06 10:23), claude/wait-poll-timeline (09-06 09:07), PR #2418 three consecutive runs on 2026-09-09 (09:50, 10:43 preflight variant, 10:51), PR #2420 (10:49, 45 s variant). Six other branches' iOS runs passed in the same window, so it is per-runner cold state, not a fleet outage.
Cause
Two cold-toolchain probes carry budgets below the first-exec stall of Apple's signature verification on a fresh host (the first xcodebuild/xcrun/large-binary exec after boot blocks roughly 18 to 19 s at 0% CPU while syspolicyd scans; the second call is instant):
packages/platform-apple/src/snapshot-source/cache-identity.ts:87:toolOutputrunsxcodebuild -version,sw_vers,uname,xcrun --sdk iphonesimulator --show-sdk-versionwithtimeoutMs: Math.min(10_000, remaining), under a 120 s overall deadline set by the verify script.packages/platform-apple/src/runner/runner-cache-metadata.ts:27:TOOLCHAIN_PROBE_TIMEOUT_MS = 5_000forxcodebuild -versionin the runner cache key.
Both budgets were sized for a warm toolchain. The overall deadlines (120 s snapshot-source, the runner preflight budget) have room; only the per-call budget trips.
Required behavior
- A toolchain identity probe tolerates one cold-start stall: per-call budget of at least 30 s bounded by the remaining deadline, and one retry after a timeout while the deadline allows, because the retry is instant once the signature check has run.
- Both sites read the same constant and carry one comment naming the stall, so the two do not drift again.
- The timeout error keeps its current text and code so existing tests and the runner-cache hint stay valid.
- Unit tests: a fake host whose first probe call exceeds the budget and whose second returns immediately passes; a host that never returns still fails at the deadline with the same error.
Non-goals
- Raising the snapshot-bridge native build timeout (
BUILD_TIMEOUT_MS) or the overall deadlines. - Retrying the compile step.
Activity
fix(apple): duplicate the cold-toolchain-probe budget instead of a sh…
fix(ios): budget cold toolchain probes for the first-exec signature s…
fix(apple): duplicate the cold-toolchain-probe budget instead of a sh…
fix(apple): bound the cold toolchain probes by the owning request budget
thymikee commentedon Sep 9, 2026
Third site, seen on PR #2421 (run 34366211699) and earlier on #2420 (10:49): the same CI step fails inside
packages/platform-apple/src/snapshot-source/native-runtime.test.ts:13-32, which compiles the bridge withxcrun --sdk iphonesimulator clang …under a hardtimeoutMs: 45_000, and the runner reportsxcrun timed out after 45000ms. That is the native compile, not the identity probe, so PR #2423 does not cover it. On a cold runner the firstxcrunhere pays the same signature stall plus the compile itself. Candidate fix, separate from #2423: budget that compile from the test's own deadline (the production path usesBUILD_TIMEOUT_MSbounded by the 120 s snapshot-source deadline) rather than a fixed 45 s, or skip the compile in the unit lane when the CI step already verified it against the clean-installed package moments earlier.fix(ios): budget cold toolchain probes for the first-exec signature s…
fix(apple): duplicate the cold-toolchain-probe budget instead of a sh…
fix(apple): bound the cold toolchain probes by the owning request budget
thymikee commentedon Sep 10, 2026
Split: the third site (the fixed 45 s clang compile budget in
native-runtime.test.ts) is now tracked as #2439. PR #2423 closes this issue for the two identity-probe sites it names.fix(ios): budget cold toolchain probes for the first-exec signature s…
fix(apple): duplicate the cold-toolchain-probe budget instead of a sh…
fix(apple): bound the cold toolchain probes by the owning request budget
fix(ios): budget cold toolchain probes for the first-exec signature s…
fix(apple): duplicate the cold-toolchain-probe budget instead of a sh…
fix(apple): bound the cold toolchain probes by the owning request budget
fix(ios): budget cold toolchain probes for the first-exec signature s…