Skip to content

Commit fed2a4c

Browse files
thymikeeclaude
andauthored
fix(perf): judge nights, samples, and failures soundly; keep logs on interrupt
- Rolling baselines take one scheduled run per UTC night (the newest usable one) and page through workflow runs until enough nights are found. Explicit --to/--from references still accept any trigger. - A run measures at least three rounds. A timing verdict needs three successful samples tonight and on each counted baseline night. - Failures get their own verdict. more-failures flags tonight failing more than every baseline night, in count and share; a faster median with worse failures is not an improvement. - The interrupt handler and the normal finish share one finalizer: save logs, stop session and daemon within a deadline, save again. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 34ec510 commit fed2a4c

13 files changed

Lines changed: 435 additions & 165 deletions

‎.github/workflows/perf-nightly.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ on:
1616
workflow_dispatch:
1717
inputs:
1818
rounds:
19-
description: 'Measured rounds per step (samples)'
19+
description: 'Measured rounds per step (samples, at least 3)'
2020
required: false
2121
default: '3'
2222
warmup:
@@ -28,7 +28,7 @@ on:
2828
required: false
2929
default: ''
3030
baseline-nights:
31-
description: 'Earlier runs in the rolling baseline'
31+
description: 'Earlier scheduled nights in the rolling baseline'
3232
required: false
3333
default: '7'
3434
baseline-branch:

‎scripts/perf/README.md‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,9 @@ The suite records a reason for each catalog command that it does not time. `ios-
1818
pnpm perf --platform ios --udid <udid> --rounds 3 --warmup 1 --out-dir .tmp/perf/local
1919
```
2020

21-
`--scenarios lifecycle,heavy-feed` runs a subset. `--app-id` changes the fixture bundle id.
21+
`--rounds` is at least 3. `--scenarios lifecycle,heavy-feed` runs a subset. `--app-id` changes the fixture bundle id.
2222
The suite uses its own temporary state directory and daemon. It stops them at the end.
23+
At the end, or on SIGINT or SIGTERM, the suite saves the logs to `raw/state/`, stops the session and daemon within a deadline, and saves the logs again.
2324

2425
## Output
2526

@@ -54,19 +55,22 @@ All times are milliseconds. Statistics use only the successful samples of a step
5455

5556
`.github/workflows/perf-nightly.yml` runs the iOS suite every night on `main`.
5657
Each run uploads `perf-ios-summary` (summary, report, comparison; 90 days) and `perf-ios-raw` (logs; 30 days).
57-
The job compares the run with up to seven earlier runs and writes the result to the job summary.
58+
The job compares the run with up to seven earlier nights and writes the result to the job summary.
59+
A night is the newest scheduled run of one UTC day that ran every scenario of the judged run.
60+
Manual dispatches are never part of the rolling baseline.
5861

5962
To fetch and compare runs on your machine, use `gh` with access to the repository:
6063

6164
```sh
6265
pnpm perf:compare # latest night against the 7 nights before it
6366
pnpm perf:compare --to 2026-10-28 # that night against the 7 nights before it
64-
pnpm perf:compare --to latest --from 2026-10-21 # latest against one night
65-
pnpm perf:compare --to latest --from 2026-10-21..2026-10-25 --out .tmp/perf/compare
67+
pnpm perf:compare --to latest --from 2026-10-21 # latest against the newest run of that day
68+
pnpm perf:compare --to latest --from 2026-10-21..2026-10-25 --out .tmp/perf/compare # every run in the range
6669
pnpm perf:compare --to .tmp/perf/local # a local run against recent nights
6770
```
6871

6972
A reference is `latest`, a run id, a `YYYY-MM-DD` date, or a local output path.
73+
An explicit `--to` or `--from` reference accepts runs of any trigger, so it can compare dispatches and local runs.
7074
Downloads stay in `.tmp/perf-nightly/<run-id>/`.
7175
Do not judge a local run against CI nights. The comparison warns when the device, Xcode, or CPU differs.
7276

@@ -81,7 +85,19 @@ A step is a `regression` or an `improvement` only when its change is more than a
8185
- Two times the coefficient of variation of the run divided by √n.
8286
- 50 ms.
8387

84-
The rolling baseline skips runs that did not run every scenario of the judged run, such as a partial dispatch.
85-
Fewer than three baseline nights give `insufficient-baseline`.
86-
`new-failures` means that the step failed and no baseline night failed. `fixed` means the reverse.
88+
Each step gets a timing verdict and a failure verdict.
89+
A timing verdict uses only successful samples:
90+
91+
- `insufficient-samples`: the judged run has fewer than three successful samples of the step.
92+
- `insufficient-baseline`: fewer than three baseline nights have three successful samples of the step. Nights with fewer do not count.
93+
94+
The failure verdict compares failed samples with the worst baseline night:
95+
96+
- `new-failures`: the step failed and no baseline night failed.
97+
- `more-failures`: the step failed more often than on every baseline night, both in count and in share of its samples.
98+
- `fixed`: the step did not fail and every baseline night failed.
99+
100+
When failures are new or more, a faster median is `insufficient-samples`, not `improvement`.
101+
The samples that still pass do not show that the step got faster.
102+
Regressions and worse failures are workflow warnings. The job stays green.
87103
The thresholds are in `compare.ts`.

‎scripts/perf/cli.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,11 @@ function jsonOk(json: unknown): boolean {
3333

3434
// Invoke the built CLI once. `args` includes the command + positionals + dash-flags;
3535
// `baseFlags` carries the isolation + device flags shared by every call.
36-
export function invokeCli(args: string[], baseFlags: string[]): CliResult {
36+
export function invokeCli(
37+
args: string[],
38+
baseFlags: string[],
39+
options: { timeoutMs?: number } = {},
40+
): CliResult {
3741
const full = [...CLI_ARGV, ...args, ...baseFlags, '--json'];
3842
const t0 = performance.now();
3943
let stdout = '';
@@ -46,6 +50,7 @@ export function invokeCli(args: string[], baseFlags: string[]): CliResult {
4650
cwd: REPO_ROOT,
4751
maxBuffer: MAX_BUFFER,
4852
allowFailure: true,
53+
timeoutMs: options.timeoutMs,
4954
});
5055
stdout = r.stdout;
5156
stderr = r.stderr;

‎scripts/perf/compare-cli.ts‎

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@ import path from 'node:path';
33
import { compareRuns, renderComparison, type Comparison } from './compare.ts';
44
import { REPO_ROOT } from './config.ts';
55
import {
6-
baselineCandidates,
76
coversTarget,
87
fetchSummary,
98
listRuns,
109
readLocalSummary,
1110
resolveRunRef,
11+
selectNightlyBaseline,
1212
selectRange,
1313
SUMMARY_ARTIFACT,
1414
type NightlySource,
@@ -24,9 +24,9 @@ A <ref> is latest, a workflow run id, a YYYY-MM-DD date (the newest run that UTC
2424
path to a summary.json or a \`pnpm perf\` output directory.
2525
2626
--to <ref> Run to judge (default: latest).
27-
--from <ref> Baseline run, or an inclusive range <ref>..<ref>. Without it the baseline is
28-
the --nights runs before --to.
29-
--nights <n> Rolling baseline size (default: 7).
27+
--from <ref> Baseline run, or an inclusive range <ref>..<ref> of runs of any trigger.
28+
Without it the baseline is the --nights scheduled nights before --to.
29+
--nights <n> Rolling baseline size in UTC nights, one scheduled run each (default: 7).
3030
--branch <name> Branch whose nightly runs are listed (default: main).
3131
--repo <owner/name> Repository (default: $GITHUB_REPOSITORY or callstack/agent-device).
3232
--workflow <file> Workflow file (default: perf-nightly.yml).
@@ -117,7 +117,7 @@ type Loaded = { summary: PerfSummary; label: string; run?: WorkflowRun };
117117
function main(): void {
118118
const options = parse(process.argv.slice(2));
119119
let runs: WorkflowRun[] | undefined;
120-
const allRuns = (): WorkflowRun[] => (runs ??= listRuns(options.source));
120+
const allRuns = (): WorkflowRun[] => (runs ??= [...listRuns(options.source)]);
121121
const runLabel = (run: WorkflowRun) => `run ${run.id} (${run.createdAt.slice(0, 10)})`;
122122
const load = (ref: string): Loaded => {
123123
if (fs.existsSync(ref))
@@ -131,34 +131,31 @@ function main(): void {
131131
if (!summary) throw new Error(`Run ${run.id} has no ${options.source.artifact} artifact.`);
132132
return { summary, label: runLabel(run), run };
133133
};
134-
const loadRuns = (
135-
candidates: WorkflowRun[],
136-
limit: number,
137-
accept: (summary: PerfSummary) => boolean = () => true,
138-
): Loaded[] => {
139-
const loaded: Loaded[] = [];
140-
for (const run of candidates) {
141-
if (loaded.length >= limit) break;
134+
const loadRuns = (candidates: WorkflowRun[]): Loaded[] =>
135+
candidates.flatMap((run) => {
142136
const summary = fetchSummary(options.source, run);
143-
if (summary && accept(summary)) loaded.push({ summary, label: runLabel(run), run });
144-
}
145-
return loaded;
146-
};
137+
return summary ? [{ summary, label: runLabel(run), run }] : [];
138+
});
147139

148140
const target = load(options.to);
149141
const exclude = new Set([...options.exclude, ...(target.run ? [target.run.id] : [])]);
150142
let baseline: Loaded[];
151143
if (options.from?.includes('..')) {
152144
const [fromRef, toRef] = options.from.split('..');
153145
const range = selectRange(allRuns(), fromRef!, toRef!).filter((run) => !exclude.has(run.id));
154-
baseline = loadRuns(range, range.length);
146+
baseline = loadRuns(range);
155147
} else if (options.from) {
156148
baseline = [load(options.from)];
157149
} else {
158150
const before = target.run?.createdAt ?? target.summary.startedAt;
159-
baseline = loadRuns(baselineCandidates(allRuns(), before, exclude), options.nights, (summary) =>
160-
coversTarget(summary, target.summary),
161-
);
151+
baseline = selectNightlyBaseline(
152+
runs ?? listRuns(options.source),
153+
{ before, exclude, nights: options.nights },
154+
(run) => {
155+
const summary = fetchSummary(options.source, run);
156+
return summary && coversTarget(summary, target.summary) ? summary : undefined;
157+
},
158+
).map(({ run, value }) => ({ summary: value, label: runLabel(run), run }));
162159
}
163160

164161
const comparison = compareRuns(target, baseline);
@@ -178,12 +175,15 @@ function publishToGitHub(comparison: Comparison, markdown: string): void {
178175
const summaryPath = process.env.GITHUB_STEP_SUMMARY;
179176
if (summaryPath) fs.appendFileSync(summaryPath, `${markdown}\n`);
180177
for (const row of comparison.rows) {
181-
if (row.verdict === 'regression' || row.verdict === 'new-failures') {
182-
const change =
183-
row.verdict === 'regression'
184-
? `median ${row.target?.median.toFixed(0)} ms vs ${row.baseline?.median.toFixed(0)} ms baseline`
185-
: `${row.target?.failures}/${row.target?.samples} samples failed`;
186-
process.stdout.write(`::warning title=perf ${row.verdict}::${row.id}: ${change}\n`);
178+
if (row.timing === 'regression') {
179+
process.stdout.write(
180+
`::warning title=perf regression::${row.id}: median ${row.target?.median.toFixed(0)} ms vs ${row.baseline?.median.toFixed(0)} ms baseline\n`,
181+
);
182+
}
183+
if (row.failures === 'new-failures' || row.failures === 'more-failures') {
184+
process.stdout.write(
185+
`::warning title=perf ${row.failures}::${row.id}: ${row.target?.failures}/${row.target?.samples} samples failed\n`,
186+
);
187187
}
188188
}
189189
}

‎scripts/perf/compare.test.ts‎

Lines changed: 36 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ test('flags a slowdown that exceeds a quiet baseline and the relative floor', ()
2626
[{ id: STEP, daemonMs: [1450, 1500, 1550] }],
2727
[flat(1000), flat(1010), flat(990)],
2828
);
29-
assert.equal(result.verdict, 'regression');
29+
assert.equal(result.timing, 'regression');
3030
assert.equal(result.metric, 'daemon');
3131
assert.equal(result.baseline?.median, 1000);
3232
assert.equal(result.deltaMs, 500);
@@ -38,37 +38,63 @@ test('does not flag the same slowdown when nightly medians already swing that mu
3838
[{ id: STEP, daemonMs: [1450, 1500, 1550] }],
3939
[flat(500), flat(1000), flat(1500), flat(700), flat(1300)],
4040
);
41-
assert.equal(result.verdict, 'unchanged');
41+
assert.equal(result.timing, 'unchanged');
4242
// 3 × scaled MAD: deviations from 1000 are 0, 300, 300, 500, 500.
4343
assert.ok(Math.abs(result.threshold! - (3 * 1.4826 * 300) / 1000) < 1e-9);
4444
});
4545

4646
test('does not flag a large relative change below the absolute floor', () => {
47-
assert.equal(row(flat(60), [flat(40), flat(40), flat(40)]).verdict, 'unchanged');
47+
assert.equal(row(flat(60), [flat(40), flat(40), flat(40)]).timing, 'unchanged');
4848
});
4949

5050
test('flags a speedup beyond noise as an improvement', () => {
51-
assert.equal(row(flat(500), [flat(1000), flat(1010), flat(990)]).verdict, 'improvement');
51+
assert.equal(row(flat(500), [flat(1000), flat(1010), flat(990)]).timing, 'improvement');
52+
});
53+
54+
test('withholds a timing verdict when tonight or a baseline night has fewer than three successes', () => {
55+
const one = (ms: number): StepFixture[] => [{ id: STEP, daemonMs: [ms] }];
56+
const fromOneSample = row(one(1500), [one(1000), one(1000), one(1000)]);
57+
assert.equal(fromOneSample.timing, 'insufficient-samples');
58+
assert.equal(fromOneSample.baseline?.timedNights, 0);
59+
// Three successes tonight still need three nights that each had three successes.
60+
const thinNights = row(flat(1500), [flat(1000), flat(1000), one(1000)]);
61+
assert.equal(thinNights.timing, 'insufficient-baseline');
62+
assert.equal(thinNights.baseline?.timedNights, 2);
63+
assert.equal(thinNights.deltaMs, 500);
5264
});
5365

5466
test('withholds a timing verdict until enough nights exist', () => {
5567
const result = row(flat(1500), [flat(1000), flat(1000)]);
56-
assert.equal(result.verdict, 'insufficient-baseline');
68+
assert.equal(result.timing, 'insufficient-baseline');
5769
assert.equal(result.deltaRelative, 0.5);
5870
});
5971

6072
test('reports failures that no baseline night had, and failures that went away', () => {
6173
const clean = [flat(1000), flat(1000), flat(1000)];
62-
assert.equal(row(flat(1000, { failures: 1 }), clean).verdict, 'new-failures');
74+
assert.equal(row(flat(1000, { failures: 1 }), clean).failures, 'new-failures');
6375
const failing = [
6476
flat(1000, { failures: 1 }),
6577
flat(1000, { failures: 2 }),
6678
flat(1000, { failures: 1 }),
6779
];
68-
assert.equal(row(flat(1000), failing).verdict, 'fixed');
69-
// A step that already failed on some baseline night is judged on timing instead.
80+
assert.equal(row(flat(1000), failing).failures, 'fixed');
81+
// As many failures as the worst baseline night is not a change; timing is judged as usual.
82+
const flaky = [flat(1000, { failures: 1 }), flat(1000), flat(1000)];
83+
const steady = row(flat(1000, { failures: 1 }), flaky);
84+
assert.equal(steady.failures, 'unchanged');
85+
assert.equal(steady.timing, 'unchanged');
86+
});
87+
88+
test('reports worse failures over a flaky baseline and never as an improvement', () => {
7089
const flaky = [flat(1000, { failures: 1 }), flat(1000), flat(1000)];
71-
assert.equal(row(flat(1000, { failures: 1 }), flaky).verdict, 'unchanged');
90+
const mostlyFailing = row([{ id: STEP, daemonMs: [500], failures: 2 }], flaky);
91+
assert.equal(mostlyFailing.failures, 'more-failures');
92+
assert.equal(mostlyFailing.timing, 'insufficient-samples');
93+
// Enough successes, but they are the subset that still passes, so the speedup is withheld.
94+
const faster = row(flat(500, { failures: 2 }), flaky);
95+
assert.equal(faster.failures, 'more-failures');
96+
assert.equal(faster.timing, 'insufficient-samples');
97+
assert.equal(faster.deltaMs, -500);
7298
});
7399

74100
test('falls back to wall time when any run lacks the daemon timer', () => {
@@ -86,7 +112,7 @@ test('marks steps that only one side measured', () => {
86112
[{ summary: summaryFixture([{ id: 'a/old', daemonMs: [1] }]), label: 'b' }],
87113
);
88114
assert.deepEqual(
89-
comparison.rows.map((r) => [r.id, r.verdict]),
115+
comparison.rows.map((r) => [r.id, r.timing]),
90116
[
91117
['a/new', 'new'],
92118
['a/old', 'missing'],

0 commit comments

Comments
 (0)