Skip to content

Commit f55d37e

Browse files
thymikeeclaude
thymikee
and
claude
committed
feat(perf): compare nightly iOS runs and run the suite every night
pnpm perf:compare judges one run against earlier runs: by default the seven nights before it, or a chosen run, date, range, or local output. A step's baseline is the median of its nightly medians. A change is flagged only when it is larger than 20%, three times the night-to-night scaled MAD, twice the run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged too. perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the comparison to the job summary with warnings for regressions and new failures. Android stays report-only behind a dispatch input. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 6454e27 commit f55d37e

14 files changed

Lines changed: 1088 additions & 23 deletions

‎.github/workflows/perf-nightly.yml‎

Lines changed: 80 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,57 @@
11
name: Perf Nightly
22

3-
# Parked to workflow_dispatch by #1781 A3: it only writes a report and compares nothing, so it
4-
# structurally cannot catch a regression, and iOS wall-clock medians swing up to +122%
5-
# night-to-night at n=5 with no consumer reading the report. `pnpm perf` / scripts/perf are
6-
# unaffected — still runnable on demand. Un-park once a comparison step with multi-night
7-
# smoothing (someone reading it) exists, or a perf arc needs nightly data.
3+
# Nightly iOS command performance against the fixture app, compared night over night (#1781 A3).
4+
# - Catches: a command whose daemon time moves beyond the noise of earlier nights, or that starts
5+
# failing. PR lanes check behavior, not time.
6+
# - Wanted for: tuning runner, snapshot, and settle paths while watching the effect. iOS medians
7+
# swing up to +122% between single nights at n=5, so a step is judged against the median of up to
8+
# seven earlier nights and must clear their spread, its own sampling error, 20%, and 50 ms.
9+
# - Cost: one macos-26 job of about 45 minutes a night; summaries are kept 90 days, raw logs 30.
10+
# - Delete or park when: no flagged change has been acted on for 60 nights.
11+
# Android stays report-only and runs only on request.
812

913
on:
14+
schedule:
15+
- cron: '40 2 * * *'
1016
workflow_dispatch:
1117
inputs:
1218
rounds:
13-
description: 'Measured rounds per command (samples)'
19+
description: 'Measured rounds per step (samples)'
1420
required: false
1521
default: '3'
22+
warmup:
23+
description: 'Warmup rounds dropped before measuring'
24+
required: false
25+
default: '1'
26+
scenarios:
27+
description: 'Comma-separated iOS scenario ids (empty runs all)'
28+
required: false
29+
default: ''
30+
baseline-nights:
31+
description: 'Earlier runs in the rolling baseline'
32+
required: false
33+
default: '7'
34+
baseline-branch:
35+
description: 'Branch whose runs form the baseline'
36+
required: false
37+
default: 'main'
38+
android:
39+
description: 'Also run the report-only Android Settings tour'
40+
type: boolean
41+
required: false
42+
default: true
1643

1744
permissions:
1845
contents: read
1946
actions: read
2047

2148
concurrency:
22-
group: ci-${{ github.workflow }}-${{ github.ref }}
23-
cancel-in-progress: true
49+
group: perf-nightly-${{ github.ref }}
50+
cancel-in-progress: false
2451

2552
env:
2653
PERF_ROUNDS: ${{ github.event.inputs.rounds || '3' }}
54+
PERF_WARMUP: ${{ github.event.inputs.warmup || '1' }}
2755

2856
jobs:
2957
perf-ios:
@@ -52,6 +80,13 @@ jobs:
5280
with:
5381
gate: build
5482

83+
- name: Boot iOS test simulator
84+
id: ios-simulator
85+
uses: ./.github/actions/boot-ios-test-simulator
86+
with:
87+
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
88+
preferred-device-name: iPhone 17 Pro
89+
5590
- name: Restore and build iOS XCTest runner
5691
uses: ./.github/actions/setup-apple-runner-build
5792
with:
@@ -62,13 +97,6 @@ jobs:
6297
xcuitest-platform: ios
6398
xcuitest-destination: generic/platform=iOS Simulator
6499

65-
- name: Boot iOS test simulator
66-
id: ios-simulator
67-
uses: ./.github/actions/boot-ios-test-simulator
68-
with:
69-
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
70-
preferred-device-name: iPhone 17 Pro
71-
72100
- name: Install the fixture app
73101
id: fixture-app
74102
uses: ./.github/actions/setup-fixture-app
@@ -88,22 +116,53 @@ jobs:
88116
UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
89117
APP_ID: ${{ steps.fixture-app.outputs.app-id }}
90118
APP_PATH: ${{ steps.fixture-app.outputs.app-path }}
119+
SCENARIOS: ${{ github.event.inputs.scenarios }}
91120
run: |
92121
pnpm perf --platform ios --udid "$UDID" \
93122
--app-id "$APP_ID" --app-path "$APP_PATH" \
94-
--rounds "$PERF_ROUNDS" --warmup 1 \
123+
--rounds "$PERF_ROUNDS" --warmup "$PERF_WARMUP" \
124+
${SCENARIOS:+--scenarios "$SCENARIOS"} \
95125
--out-dir "$GITHUB_WORKSPACE/perf-results"
96126
97-
- name: Upload iOS perf report
98-
if: always()
127+
- name: Compare with earlier nights
128+
if: ${{ !cancelled() && hashFiles('perf-results/summary.json') != '' }}
129+
env:
130+
GH_TOKEN: ${{ github.token }}
131+
BASELINE_BRANCH: ${{ github.event.inputs.baseline-branch || 'main' }}
132+
BASELINE_NIGHTS: ${{ github.event.inputs.baseline-nights || '7' }}
133+
run: |
134+
pnpm perf:compare --to perf-results \
135+
--branch "$BASELINE_BRANCH" \
136+
--nights "$BASELINE_NIGHTS" \
137+
--exclude-run "$GITHUB_RUN_ID" \
138+
--out perf-results --github
139+
echo "Full report: the perf-ios-summary artifact holds report.md and summary.json." >> "$GITHUB_STEP_SUMMARY"
140+
141+
- name: Upload iOS perf summary
142+
if: ${{ !cancelled() && hashFiles('perf-results/summary.json') != '' }}
99143
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
100144
with:
101-
name: perf-ios
102-
path: perf-results/
145+
name: perf-ios-summary
146+
path: |
147+
perf-results/summary.json
148+
perf-results/report.md
149+
perf-results/comparison.json
150+
perf-results/comparison.md
151+
retention-days: 90
152+
if-no-files-found: error
153+
154+
- name: Upload iOS perf raw logs
155+
if: ${{ always() }}
156+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
157+
with:
158+
name: perf-ios-raw
159+
path: perf-results/raw/
160+
retention-days: 30
103161
if-no-files-found: warn
104162

105163
perf-android:
106164
name: Android Command Perf
165+
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.android == 'true' }}
107166
runs-on: ubuntu-latest
108167
timeout-minutes: 80
109168
env:
@@ -134,7 +193,7 @@ jobs:
134193
adb -s emulator-5554 shell settings put global window_animation_scale 0 || true;
135194
adb -s emulator-5554 shell settings put global transition_animation_scale 0 || true;
136195
adb -s emulator-5554 shell settings put global animator_duration_scale 0 || true;
137-
pnpm perf --platform android --serial emulator-5554 --n "$PERF_ROUNDS" --warmup 1 --out-dir "$GITHUB_WORKSPACE/perf-results"
196+
pnpm perf --platform android --serial emulator-5554 --n "$PERF_ROUNDS" --warmup "$PERF_WARMUP" --out-dir "$GITHUB_WORKSPACE/perf-results"
138197
139198
- name: Upload Android perf report
140199
if: always()

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@
131131
"test:ios-snapshot-differential": "node --experimental-strip-types scripts/swift-toolchain-tmpdir.ts swift test --package-path apple/snapshot-presentation && node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/ios-snapshot-differential.test.ts",
132132
"size": "node scripts/size-report.mjs",
133133
"perf": "node --experimental-strip-types scripts/perf/run.ts",
134+
"perf:compare": "node --experimental-strip-types scripts/perf/compare-cli.ts",
134135
"bench:ios-snapshot": "node --experimental-strip-types scripts/ios-snapshot-benchmark/run.ts",
135136
"bench:ios-snapshot:deep-button": "node --experimental-strip-types scripts/ios-snapshot-benchmark/deep-button.ts",
136137
"bench:ios-snapshot:evidence": "node --experimental-strip-types scripts/ios-snapshot-benchmark/evidence.ts",

‎scripts/perf/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,39 @@ All times are milliseconds. Statistics use only the successful samples of a step
4949
| backends | The `snapshotQuality` backend, state, and reason code, for example `private-ax/recovered/ax-rejected`. |
5050
| fallbacks | Degraded paths: snapshot-source errors, truncated recovery, and warn-level phases. |
5151
| `fail` | The number of failed samples. `setup_failed` and `before_failed` mean that the untimed preparation failed. `daemon_stuck` means that an earlier request outlived its timeout; the suite restarts the daemon before the next scenario. `expectation_failed` means that the screen no longer has the shape that the step requires, for example the heavy-feed depth. |
52+
53+
## Nightly results
54+
55+
`.github/workflows/perf-nightly.yml` runs the iOS suite every night on `main`.
56+
Each run uploads `perf-ios-summary` (summary, report, comparison; 90 days) and `perf-ios-raw` (logs; 30 days).
57+
The job compares the run with up to seven earlier runs and writes the result to the job summary.
58+
59+
To fetch and compare runs on your machine, use `gh` with access to the repository:
60+
61+
```sh
62+
pnpm perf:compare # latest night against the 7 nights before it
63+
pnpm perf:compare --to 2026-10-28 # that night against the 7 nights before it
64+
pnpm perf:compare --to latest --from 2026-10-21 # latest against one night
65+
pnpm perf:compare --to latest --from 2026-10-21..2026-10-25 --out .tmp/perf/compare
66+
pnpm perf:compare --to .tmp/perf/local # a local run against recent nights
67+
```
68+
69+
A reference is `latest`, a run id, a `YYYY-MM-DD` date, or a local output path.
70+
Downloads stay in `.tmp/perf-nightly/<run-id>/`.
71+
Do not judge a local run against CI nights. The comparison warns when the device, Xcode, or CPU differs.
72+
73+
## Comparison rule
74+
75+
The baseline of a step is the median of its nightly medians.
76+
The noise of a step is the scaled median absolute deviation of those medians.
77+
A step is a `regression` or an `improvement` only when its change is more than all of these:
78+
79+
- 20% of the baseline.
80+
- Three times the night-to-night noise.
81+
- Two times the coefficient of variation of the run divided by √n.
82+
- 50 ms.
83+
84+
The rolling baseline skips runs that did not run every scenario of the judged run, such as a partial dispatch.
85+
Fewer than three baseline nights give `insufficient-baseline`.
86+
`new-failures` means that the step failed and no baseline night failed. `fixed` means the reverse.
87+
The thresholds are in `compare.ts`.

0 commit comments

Comments
 (0)