Skip to content

Commit 7606f85

Browse files
committed
perf(ci): overlap iOS simulator boot with the native build
1 parent f8eddcb commit 7606f85

5 files changed

Lines changed: 260 additions & 16 deletions

File tree

‎.github/actions/boot-ios-test-simulator/action.yml‎

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,15 @@ inputs:
1414
required: false
1515
default: '300'
1616

17+
wait-for-boot:
18+
description: 'Wait for boot completion before returning'
19+
required: false
20+
default: 'true'
21+
1722
outputs:
23+
boot-timeout-seconds:
24+
description: 'Boot wait budget for a deferred readiness check'
25+
value: ${{ inputs.boot-timeout-seconds }}
1826
simulator-udid:
1927
description: 'UDID of the booted iOS simulator'
2028
value: ${{ steps.boot.outputs.simulator-udid }}
@@ -24,8 +32,14 @@ runs:
2432
steps:
2533
- name: Resolve and boot iOS test simulator
2634
id: boot
35+
env:
36+
WAIT_FOR_BOOT: ${{ inputs.wait-for-boot }}
2737
run: |
2838
set -euo pipefail
39+
if [[ "$WAIT_FOR_BOOT" != 'true' && "$WAIT_FOR_BOOT" != 'false' ]]; then
40+
echo 'wait-for-boot must be true or false' >&2
41+
exit 1
42+
fi
2943
RUNTIME_VERSION="${{ inputs.runtime-version }}"
3044
RUNTIME_TOKEN="SimRuntime.iOS-${RUNTIME_VERSION//./-}"
3145
export RUNTIME_TOKEN
@@ -58,15 +72,7 @@ runs:
5872
echo "simulator-udid=$UDID" >> "$GITHUB_OUTPUT"
5973
xcrun simctl shutdown all || true
6074
xcrun simctl boot "$UDID" || true
61-
TIMEOUT_SECONDS="${{ inputs.boot-timeout-seconds }}"
62-
set +e
63-
perl -e 'alarm shift; exec @ARGV' "$TIMEOUT_SECONDS" xcrun simctl bootstatus "$UDID" -b
64-
BOOTSTATUS_EXIT="$?"
65-
set -e
66-
if [ "$BOOTSTATUS_EXIT" -eq 142 ]; then
67-
echo "Timed out waiting ${TIMEOUT_SECONDS}s for simulator $UDID to boot" >&2
68-
xcrun simctl list devices || true
69-
exit 1
75+
if [ "$WAIT_FOR_BOOT" = 'true' ]; then
76+
bash scripts/wait-for-ios-simulator.sh "$UDID" "${{ inputs.boot-timeout-seconds }}"
7077
fi
71-
exit "$BOOTSTATUS_EXIT"
7278
shell: bash

‎.github/actions/smoke-ios-run/action.yml‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,13 @@ runs:
3232
mkdir -p .tmp
3333
echo "$FRONTMOST" > .tmp/ios-host-focus-before
3434
shell: bash
35+
- name: Boot iOS test simulator
36+
id: ios-simulator
37+
uses: ./.github/actions/boot-ios-test-simulator
38+
with:
39+
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
40+
preferred-device-name: iPhone 17 Pro
41+
wait-for-boot: 'false'
3542
- name: Restore and build iOS XCTest runner
3643
uses: ./.github/actions/setup-apple-runner-build
3744
with:
@@ -41,12 +48,12 @@ runs:
4148
gate: swift-runner-ios
4249
xcuitest-platform: ios
4350
xcuitest-destination: generic/platform=iOS Simulator
44-
- name: Boot iOS test simulator
45-
id: ios-simulator
46-
uses: ./.github/actions/boot-ios-test-simulator
47-
with:
48-
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
49-
preferred-device-name: iPhone 17 Pro
51+
- name: Wait for iOS test simulator
52+
run: |
53+
bash scripts/wait-for-ios-simulator.sh \
54+
"${{ steps.ios-simulator.outputs.simulator-udid }}" \
55+
"${{ steps.ios-simulator.outputs.boot-timeout-seconds }}"
56+
shell: bash
5057
- name: Run iOS runner platform checks
5158
id: ios-xctest
5259
if: steps.xctest-impact.outputs.run != 'false'
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# iOS simulator boot and native build overlap
2+
3+
## Change
4+
5+
The iOS smoke lane starts its selected simulator before the generic native build.
6+
The build runs while CoreSimulator completes boot.
7+
A required boot-status barrier runs before the XCTest platform checks and public CLI preflight.
8+
The barrier keeps the existing 300-second wait limit and propagates native failures.
9+
Other users of the boot action still wait before the action returns.
10+
No test selection or runtime policy changes.
11+
12+
## Hosted baseline
13+
14+
[Apple run 39287002947](https://github.com/callstack/agent-device/actions/runs/39287002947/job/118176526594), source `14f32fb218ac9e5608012bf6a5805652dc61f7fc`, passed in 23m49s.
15+
The native runner preparation took 74.238s.
16+
The subsequent simulator preparation took 109.032s.
17+
These phases ran serially.
18+
Their times show an overlap opportunity, not a promised hosted saving.
19+
20+
## Local comparison
21+
22+
The native sources are from base `f8eddcbda35b4b1ee4e28de963925c77d77eab56`.
23+
Only workflow ordering and the shared wait script differ.
24+
The host uses Xcode 26.2 and iOS 26.2 on arm64.
25+
Each sample creates a fresh iPhone 17 Pro simulator and a separate DerivedData directory.
26+
The build uses `scripts/build-xcuitest-apple.sh`, unit-test inclusion, arm64, and `generic/platform=iOS Simulator`.
27+
The serial sample builds before it starts boot.
28+
The overlap sample starts boot before it builds.
29+
Both samples then require `scripts/wait-for-ios-simulator.sh <udid> 300` to succeed.
30+
Creation, shutdown and deletion are outside the measured preparation interval.
31+
All four native builds and boot barriers pass.
32+
33+
| Order | Native build | Preparation wall time |
34+
| --- | ---: | ---: |
35+
| 0-serial | 12.834s | 41.556s |
36+
| 1-overlap | 17.644s | 29.006s |
37+
| 2-overlap | 17.687s | 30.299s |
38+
| 3-serial | 15.872s | 39.922s |
39+
40+
The two-sample means are 40.739s serial and 29.653s overlapping.
41+
The observed saving is 11.086s (27.2%) for preparation on this host.
42+
Overlap makes each build slower, but reduces the combined preparation time in both comparisons.
43+
These are local samples with evolving system caches, not controlled hosted speed evidence.
44+
The hosted lane uses a different Xcode and simulator image.
45+
Hosted Apple CI must confirm correctness and total job cost.
46+
47+
## Failure controls and cleanup
48+
49+
The focused workflow suite passes 40 tests.
50+
It executes the boot action with fake platform tools.
51+
It verifies deferred boot, default blocking behavior, invalid-policy rejection, native failure propagation, and the mandatory readiness barrier before XCTest.
52+
An injected alarm exit verifies the timeout branch without sleeping through a timeout in unit tests.
53+
The real native comparisons above use the actual bounded wait.
54+
55+
All four owned simulators were shut down and deleted.
56+
Other devices were not changed.
57+
Commands, native build logs, boot logs and results remain under `/private/tmp/pr-apple-boot-benchmark`.

‎scripts/__tests__/apple-ci-impact.test.ts‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -547,3 +547,158 @@ test('a shallow PR merge still yields a known change set for all Apple selectors
547547
fs.rmSync(root, { recursive: true, force: true });
548548
}
549549
});
550+
551+
test('iOS boot overlaps the native build and readiness remains mandatory before XCTest', () => {
552+
const document = parse(
553+
fs.readFileSync(path.join(repoRoot, '.github/actions/smoke-ios-run/action.yml'), 'utf8'),
554+
) as { runs: { steps: AppleStep[] } };
555+
const steps = document.runs.steps;
556+
const boot = steps.findIndex((step) => step.id === 'ios-simulator');
557+
const build = steps.findIndex(
558+
(step) => step.uses === './.github/actions/setup-apple-runner-build',
559+
);
560+
const wait = steps.findIndex((step) => step.run?.includes('scripts/wait-for-ios-simulator.sh'));
561+
const tests = steps.findIndex((step) => step.id === 'ios-xctest');
562+
expect(boot).toBeGreaterThanOrEqual(0);
563+
expect(build).toBeGreaterThan(boot);
564+
expect(wait).toBeGreaterThan(build);
565+
expect(tests).toBeGreaterThan(wait);
566+
expect(steps[boot]!.with?.['wait-for-boot']).toBe('false');
567+
expect(steps[wait]!.run).toContain('steps.ios-simulator.outputs.simulator-udid');
568+
expect(steps[wait]!.run).toContain('steps.ios-simulator.outputs.boot-timeout-seconds');
569+
expect(steps[wait]!.if).toBeUndefined();
570+
expect(steps[wait]!['continue-on-error']).not.toBe(true);
571+
});
572+
573+
function simulatorBootFixture(mode: 'ready' | 'failed' | 'stalled') {
574+
const directory = mkdtempForTestSync('ios-boot-wait-');
575+
const calls = path.join(directory, 'calls.ndjson');
576+
fs.writeFileSync(
577+
path.join(directory, 'xcrun'),
578+
`#!${process.execPath}
579+
const fs = require('node:fs');
580+
const args = process.argv.slice(2);
581+
fs.appendFileSync(process.env.FIXTURE_CALLS, JSON.stringify(args) + '\\n');
582+
if (args.includes('list')) console.log(JSON.stringify({ devices: { 'com.apple.CoreSimulator.SimRuntime.iOS-26-2': [{ udid: 'owned-ios', name: 'iPhone 17 Pro', isAvailable: true, state: 'Shutdown' }] } }));
583+
if (args.includes('bootstatus')) {
584+
if (process.env.FIXTURE_BOOT_MODE === 'failed') process.exit(7);
585+
if (process.env.FIXTURE_BOOT_MODE === 'stalled') setInterval(() => {}, 1000);
586+
}
587+
`,
588+
{ mode: 0o755 },
589+
);
590+
if (mode === 'stalled')
591+
fs.writeFileSync(
592+
path.join(directory, 'perl'),
593+
`#!${process.execPath}
594+
const fs = require('node:fs');
595+
fs.appendFileSync(process.env.FIXTURE_CALLS, JSON.stringify(['alarm', ...process.argv.slice(2)]) + '\\n');
596+
process.exit(142);
597+
`,
598+
{ mode: 0o755 },
599+
);
600+
return {
601+
directory,
602+
env: {
603+
...process.env,
604+
PATH: `${directory}:${process.env.PATH}`,
605+
FIXTURE_CALLS: calls,
606+
FIXTURE_BOOT_MODE: mode,
607+
},
608+
calls: () =>
609+
fs.existsSync(calls)
610+
? fs
611+
.readFileSync(calls, 'utf8')
612+
.trim()
613+
.split('\n')
614+
.filter(Boolean)
615+
.map((line) => JSON.parse(line) as string[])
616+
: [],
617+
};
618+
}
619+
620+
function bootActionRun(waitForBoot?: string, mode: 'ready' | 'failed' = 'ready') {
621+
const fixture = simulatorBootFixture(mode);
622+
const doc = parse(
623+
fs.readFileSync(
624+
path.join(repoRoot, '.github/actions/boot-ios-test-simulator/action.yml'),
625+
'utf8',
626+
),
627+
) as { inputs: Record<string, { default?: string }>; runs: { steps: AppleStep[] } };
628+
const inputs = {
629+
'runtime-version': '26.2',
630+
'preferred-device-name': 'iPhone 17 Pro',
631+
'boot-timeout-seconds': '2',
632+
};
633+
const script = doc.runs.steps[0]!.run!.replaceAll(
634+
/\$\{\{ inputs\.([\w-]+) \}\}/g,
635+
(_whole, key: keyof typeof inputs) => inputs[key],
636+
);
637+
const result = spawnSync('/bin/bash', ['-c', script], {
638+
cwd: repoRoot,
639+
env: {
640+
...fixture.env,
641+
WAIT_FOR_BOOT: waitForBoot ?? doc.inputs['wait-for-boot']!.default,
642+
GITHUB_OUTPUT: path.join(fixture.directory, 'outputs'),
643+
},
644+
encoding: 'utf8',
645+
timeout: 10_000,
646+
});
647+
return { result, calls: fixture.calls() };
648+
}
649+
650+
test('deferred boot starts the selected simulator without waiting on its readiness', () => {
651+
const { result, calls } = bootActionRun('false', 'failed');
652+
expect(result.error).toBeUndefined();
653+
expect(result.status, result.stderr).toBe(0);
654+
expect(calls).toContainEqual(['simctl', 'boot', 'owned-ios']);
655+
expect(calls.some((args) => args.includes('bootstatus'))).toBe(false);
656+
});
657+
658+
test('existing boot action callers still wait and propagate a bootstatus failure', () => {
659+
const { result, calls } = bootActionRun(undefined, 'failed');
660+
expect(result.error).toBeUndefined();
661+
expect(result.status, result.stderr).toBe(7);
662+
expect(calls).toContainEqual(['simctl', 'bootstatus', 'owned-ios', '-b']);
663+
});
664+
665+
test('invalid deferred-boot policy fails before changing simulator state', () => {
666+
const { result, calls } = bootActionRun('unexpected');
667+
expect(result.status).toBe(1);
668+
expect(calls).toEqual([]);
669+
});
670+
671+
test.each([
672+
['ready', '2', 0],
673+
['failed', '2', 7],
674+
['stalled', '1', 1],
675+
['ready', '0', 1],
676+
] as const)(
677+
'deferred readiness barrier keeps native outcome and deadline: %s, %s seconds',
678+
(mode, timeout, expected) => {
679+
const fixture = simulatorBootFixture(mode);
680+
const result = spawnSync(
681+
'/bin/bash',
682+
[path.join(repoRoot, 'scripts/wait-for-ios-simulator.sh'), 'owned-ios', timeout],
683+
{ env: fixture.env, encoding: 'utf8', timeout: 5000 },
684+
);
685+
expect(result.error).toBeUndefined();
686+
expect(result.status, result.stderr).toBe(expected);
687+
if (mode === 'stalled')
688+
expect(result.stderr).toContain('Timed out waiting 1s for simulator owned-ios');
689+
if (timeout === '0') expect(fixture.calls()).toEqual([]);
690+
else if (mode === 'stalled')
691+
expect(fixture.calls()).toContainEqual([
692+
'alarm',
693+
'-e',
694+
'alarm shift; exec @ARGV',
695+
timeout,
696+
'xcrun',
697+
'simctl',
698+
'bootstatus',
699+
'owned-ios',
700+
'-b',
701+
]);
702+
else expect(fixture.calls()).toContainEqual(['simctl', 'bootstatus', 'owned-ios', '-b']);
703+
},
704+
);

‎scripts/wait-for-ios-simulator.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
UDID="${1:?simulator UDID is required}"
5+
TIMEOUT_SECONDS="${2:?boot timeout is required}"
6+
if [[ ! "$TIMEOUT_SECONDS" =~ ^[1-9][0-9]*$ ]]; then
7+
echo 'Boot timeout must be a positive integer' >&2
8+
exit 1
9+
fi
10+
set +e
11+
perl -e 'alarm shift; exec @ARGV' "$TIMEOUT_SECONDS" xcrun simctl bootstatus "$UDID" -b
12+
BOOTSTATUS_EXIT="$?"
13+
set -e
14+
if [ "$BOOTSTATUS_EXIT" -eq 142 ]; then
15+
echo "Timed out waiting ${TIMEOUT_SECONDS}s for simulator $UDID to boot" >&2
16+
xcrun simctl list devices || true
17+
exit 1
18+
fi
19+
exit "$BOOTSTATUS_EXIT"

0 commit comments

Comments
 (0)