Skip to content

Commit ddc0d50

Browse files
authored
test(ios): close the snapshot convergence evidence sweep at a final head (#2188) (#2750)
* fix(ios-benchmark): wait out unmounted fixture screens during setup admission The proxy leg stopped on `fixture-anchor` for every deep-linked screen it reached: the persistent client reads the anchor right after `apps.open`, and that read lands before the app mounts its first tree. Measured on the fixture app, the snapshot taken immediately after an `open --relaunch --launch-url` returns the single Application node, while a read 1.5s later returns 31 nodes with the anchor. The fresh-process CLI leg hides the same gap behind its per-read process spawn. Setup reads are untimed, so the shared admission now re-observes until the expected anchor is exposed, bounded by FIXTURE_ANCHOR_ADMISSION_BUDGET_MS, and still stops the run on the same typed reason when a fixture never exposes it. An unsuccessful observation keeps failing immediately rather than polling. Also repairs the test module's stale `FixturePreparationResult` import: the scripts directory is erased rather than typechecked, so the rename to `FixtureOperationResult` went unnoticed. * docs(evidence): record the iOS snapshot convergence final head The sweep that gated #2188 on #2199 completed at `7c434b5758`: cold/cold-cold, first-interaction, warm/relaunch with the package-size leg, and the 0/20/80 ms proxy matrix, every leg clean and free of failed samples. The four Markdown summaries ride along; the record states what the numbers prove and the deviations that bound them — target and runtime, host load, the fixture app's changed trees, and the 196-commit span behind the package-size delta. Two findings that should not travel quietly: the warm snapshot daemon cost is about double the baseline under a controlled side-by-side diagnostic, and the #2198 slice corpora come from unmerged branch heads, so they are not parity baselines for anything. * chore(gates): pin the iOS snapshot convergence final corpus `PUBLISHED_EVIDENCE` becomes `PUBLISHED_CORPORA`: the pinned evidence is now two corpora measured at two revisions on two runtimes, each carrying its own tag and evidence commit, and `PUBLISHED_EVIDENCE` derives from them. A missing file is now fetched with a hint naming the corpus that actually holds it, instead of a single tag that is wrong for one of them. Disjointness, naming, and the tag and commit each corpus is cited by are tests rather than comments. Also stops exporting the setup admission's internals, which the dead-code audit rightly flags.
1 parent 6903130 commit ddc0d50

12 files changed

Lines changed: 681 additions & 82 deletions
Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# iOS snapshot convergence final-head evidence
2+
3+
- Issues: #2188 (gate), #2199 (measurement), #2189 (baseline corpus), #1571/#1626 (corpus lineage)
4+
- Observed: 2026-09-21, 19:03–22:52 CEST (legs sequential, one subprocess-backed gate per host)
5+
- Revision: `7c434b5758` (`docs/ios-snapshot-final-evidence`, clean tree at every leg; `revision.dirty: false`)
6+
- Target: `bench-2188-final` iPhone 17 Pro Simulator, iOS 26.2, UDID `5FEADD02-98E4-4F01-861C-07003C1A3291`
7+
- Host: MacBook Pro `Mac16,8`, Apple M4 Pro, 12 cores, Xcode 26.2 (17C52), Node v26.8.1
8+
- App: `Agent Device Tester` (`com.callstack.agentdevicelab`) from the CI artifact
9+
`fingerprint.d6b4c1f5e022e54074df584e332d1e0f100c7a0f.ios` (run `35530285864`), repacked with the
10+
measured head's own JavaScript bundle
11+
- Corpora: tag `evidence/ios-snapshot/7c434b575`, evidence commit `96d4951c19`, four files listed
12+
with their SHA-256 digests in [`scripts/ios-snapshot-benchmark/evidence/README.md`](../../scripts/ios-snapshot-benchmark/evidence/README.md)
13+
- Baseline: tag `evidence/ios-snapshot/71fb2483f`, evidence commit `2d4baf461aa`, measured at
14+
`71fb2483f3` on `bench-golden-v2` (iOS 27.0)
15+
16+
## What this closes
17+
18+
#2188 was gated on #2199's final exact-head evidence sweep: the package-size delta against #2189, the
19+
conformance/fuzz/property corpus, provider contracts, the Simulator and proxy legs, the no-regrowth
20+
enforcement, and a release record at a named head. All of it now exists at `7c434b5758`.
21+
22+
The proxy leg was the last blocker, and it was blocked by the harness, not by the product. The
23+
anchor admission the harness gained at `71fb2483f3` reads the tree exactly once after `apps.open`.
24+
On this target the first tree lands later than that read: the snapshot taken immediately after an
25+
`open --relaunch --launch-url` returns the single `Application` node, while a read 1.5 s later
26+
returns 31 nodes with the `Catalog` anchor. Every deep-linked screen the leg reached stopped with
27+
`fixture-anchor`. `45e4c594a1..7c434b5758` puts a bounded wait into that untimed setup admission
28+
(`FIXTURE_ANCHOR_ADMISSION_BUDGET_MS`, 30 s, 500 ms polling) and keeps the same typed stop when a
29+
fixture never exposes its anchor. No `src/`, `packages/`, app, or CLI runtime file differs between
30+
those two commits; the measured binary is the measured head's own `pnpm build` output.
31+
32+
## Deviations from the baseline corpus
33+
34+
Stated in full, because they bound what these numbers can prove.
35+
36+
| Deviation | This corpus | Baseline corpus |
37+
| --- | --- | --- |
38+
| Simulator and runtime | `bench-2188-final`, iOS 26.2 | `bench-golden-v2`, iOS 27.0 |
39+
| Why | `bench-golden-v2`'s iOS 27.0 runtime was not installed on this host at measurement time | — |
40+
| Host load (1-min average, sampled per leg) | warm/relaunch 4.2–31.2, proxy 6.4–30.3, cold 5.3–96.7 | quiet |
41+
| Fixture app trees | catalog 31 nodes, nested-scroll 23, checkout 25, alert 26, Settings 18, inert 4 | catalog 35, nested-scroll 25, checkout 29, alert 28, Settings 18, inert 6 |
42+
| Package-size span | `71fb2483f3..7c434b5758`: 196 commits of unrelated product work | — |
43+
44+
Because the fixture trees differ, response sizes are not comparable cell for cell, and any timing
45+
delta on a screen whose tree changed is only partly attributable to the tooling. Timings are bounded
46+
observations under uncontrolled host load, not a general performance guarantee.
47+
48+
## Result — cold and cold-cold (10 samples per cell, fresh CLI process)
49+
50+
| State | Screen | Wall median | Wall median (base) | Daemon median | Daemon median (base) |
51+
| --- | --- | ---: | ---: | ---: | ---: |
52+
| cold-cold | quiet | 7,159 | 17,544 | 5,709 | 15,161 |
53+
| cold-cold | list | 7,580 | 19,170 | 5,881 | 17,191 |
54+
| cold-cold | nested-scroll | 8,091 | 19,407 | 6,374 | 17,773 |
55+
| cold-cold | alert | 7,461 | 19,448 | 5,689 | 17,436 |
56+
| cold-cold | system-surface | 6,553 | 20,235 | 4,961 | 17,974 |
57+
| cold-cold | xctest-stress | 7,273 | 20,811 | 5,761 | 18,506 |
58+
| cold | quiet | 5,743 | 6,643 | 4,409 | 5,886 |
59+
| cold | list | 5,847 | 6,851 | 4,420 | 5,897 |
60+
| cold | nested-scroll | 6,058 | 6,713 | 4,619 | 5,884 |
61+
| cold | alert | 5,706 | 6,674 | 4,310 | 5,868 |
62+
| cold | system-surface | 5,028 | 6,819 | 3,662 | 5,957 |
63+
| cold | xctest-stress | 5,679 | 6,714 | 4,323 | 5,874 |
64+
65+
Milliseconds. `cold-cold` shuts the Simulator down, clears benchmark-owned derived data, and boots it
66+
again before every sample; `cold` stops the daemon and terminates the app. Cold-cold wall medians
67+
fall by 57–68% and daemon medians by 62–72% against the baseline, measured while this host carried
68+
substantially more load than the baseline's. Cold (booted Simulator, cold runner) improves by 9–26%.
69+
70+
## Result — warm, relaunch, and first interaction
71+
72+
Warm and relaunch carry 20 samples per cell; first interaction carries 10 and has no baseline-corpus
73+
counterpart, so it is reported on its own.
74+
75+
| Screen | Warm wall | Warm wall (base) | Warm daemon | Warm daemon (base) | Relaunch wall | Relaunch wall (base) | First interaction wall |
76+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
77+
| quiet | 169 | 154 | 89 | 50 | 3,876 | 3,645 | 928 |
78+
| list | 291 | 312 | 202 | 209 | 3,888 | 4,378 | 1,436 |
79+
| nested-scroll | 250 | 187 | 167 | 83 | 3,872 | 4,313 | 741 |
80+
| alert | 224 | 207 | 139 | 104 | 3,896 | 4,370 | 1,005 |
81+
| system-surface | 208 | 210 | 130 | 105 | 3,576 | 4,917 | 1,119 |
82+
| xctest-stress | 213 | 208 | 128 | 105 | 3,868 | 4,260 | 1,302 |
83+
84+
Milliseconds. Relaunch is at or better than the baseline on five of six screens. Warm wall is within
85+
±10% of the baseline on five of six screens; warm daemon duration is uniformly higher, which is the
86+
finding below.
87+
88+
## Result — proxy transport (20 samples per screen at each added RTT)
89+
90+
The headline of the convergence work: a warm snapshot crossed the tunnel with far fewer round trips
91+
than at the baseline, so added latency costs far less.
92+
93+
| Screen | Transport | 0 ms | 20 ms | 80 ms | 0 ms (base) | 20 ms (base) | 80 ms (base) |
94+
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
95+
| quiet | fresh CLI | 184 | 293 | 556 | 233 | 1,057 | 1,801 |
96+
| quiet | persistent client | 103 | 158 | 282 | 273 | 562 | 806 |
97+
| list | fresh CLI | 304 | 415 | 691 | 431 | 1,221 | 1,328 |
98+
| list | persistent client | 214 | 280 | 408 | 254 | 719 | 729 |
99+
| nested-scroll | fresh CLI | 260 | 377 | 643 | 188 | 1,116 | 1,108 |
100+
| nested-scroll | persistent client | 179 | 240 | 381 | 138 | 590 | 616 |
101+
| alert | fresh CLI | 241 | 343 | 615 | 210 | 1,113 | 1,362 |
102+
| alert | persistent client | 148 | 203 | 351 | 108 | 622 | 637 |
103+
| system-surface | fresh CLI | 217 | 319 | 595 | 209 | 1,119 | 1,145 |
104+
| system-surface | persistent client | 128 | 183 | 330 | 105 | 537 | 638 |
105+
| xctest-stress | fresh CLI | 229 | 340 | 562 | 225 | 1,136 | 1,395 |
106+
| xctest-stress | persistent client | 136 | 197 | 314 | 127 | 612 | 619 |
107+
108+
Milliseconds, wall-clock medians. Going from 0 to 80 ms of added RTT costs 178–387 ms per snapshot
109+
here, against 475–1,568 ms at the baseline. On the quiet fixture over the fresh CLI transport that
110+
difference is about twenty RTT-bound exchanges per snapshot at the baseline against about five here;
111+
over the persistent client it is about seven against about two. Every cell completed with zero failed
112+
samples at every RTT, so the retry/timeout path is not what improved.
113+
114+
## Result — package size against #2189
115+
116+
| Measurement | `7c434b5758` | `71fb2483f3` | Delta |
117+
| --- | ---: | ---: | ---: |
118+
| Packed tarball | 1,409,306 B | 982,960 B | +43.4% |
119+
| Clean-installed package | 4,723,895 B / 543 files | 3,347,347 B / 436 files | +41.1% / +107 |
120+
| Bundled JavaScript | 3,775,024 B raw, 1,258,950 B gzip, 410 files | 2,480,947 B raw, 835,279 B gzip, 330 files | +52.2% raw, +50.7% gzip |
121+
122+
The span covers 196 commits of unrelated product work, so this delta cannot be attributed to the
123+
snapshot convergence alone; it is recorded as the size at the measured head, and the leg reproduces
124+
to the byte across independent runs (the same `1,409,306` tarball twice).
125+
126+
## Result — deterministic corpus and no-regrowth enforcement
127+
128+
| Gate | Command | Result |
129+
| --- | --- | --- |
130+
| Swift/TypeScript presentation differential | `pnpm test:ios-snapshot-differential` | pass (`swift test --package-path apple/snapshot-presentation`, plus 3 differential tests) |
131+
| Engine conformance, properties, transitions, tree | `vitest run packages/capture-kit/src/ios-snapshot-engine` | pass (10 files, 67 tests) |
132+
| Fuzz corpus replay | `vitest run scripts/fuzz/corpus-replay.test.ts` | pass (11 cases) |
133+
| Fuzz worker lane | `pnpm test:fuzz-worker` | pass (11 cases) |
134+
| Provider contracts | `pnpm test:integration:provider` | pass (66 files, 212 tests) |
135+
| No-regrowth rules R72/R73/R74 | `pnpm check:layering` | pass (`ios-snapshot-engine-ownership`, `provider-snapshot-presentation-ownership`, `snapshot-assembly-presentation-neutrality`, wired in `scripts/layering/check.ts`) |
136+
| Benchmark harness unit suite | `vitest run scripts/ios-snapshot-benchmark` | pass (17 files, 50 tests) |
137+
| Deep-button controls | inside every corpus (`deepButtonEvidence`) | the shallow-rule control exits 1 with "changed descendant was omitted by shallow observation"; the safe control exits 0 |
138+
| Affected gate | `pnpm check:affected --run` | see the PR; GitHub stays authoritative for provider integration, coverage, native builds, and device lanes |
139+
140+
## Finding — warm snapshot daemon cost is about double the baseline
141+
142+
Recorded here rather than hidden, and not explained by target, runtime, app, or load.
143+
144+
An off-corpus controlled diagnostic: same Simulator, same installed app, same ambient load
145+
(1-minute average ~21), back-to-back runs of ten warm `interactiveOnly` snapshots of the smallest
146+
fixture (4 nodes, ~3.4 KB), reading the daemon's own per-step duration.
147+
148+
| CLI under test | Durations (ms) | Median |
149+
| --- | --- | ---: |
150+
| `71fb2483f3` (baseline corpus head) | 68, 47, 46, 44, 43, 43, 43, 45, 47, 46 | 45.5 |
151+
| `7c434b5758` (this head), round 1 | 97, 86, 84, 90, 84, 86, 86, 92, 83, 82 | 86 |
152+
| `7c434b5758` (this head), round 2 | 98, 133, 96, 96, 88, 90, 87, 94, 107, 103 | 96 |
153+
154+
The published warm cells agree with the diagnostic (89–202 ms against the baseline's 50–209 ms), and
155+
a repeat at a 1-minute load average of 3.8 still measured 86 ms, so contention is not the cause. The
156+
shape is a roughly fixed ~40 ms per snapshot rather than work proportional to tree size. Candidates
157+
in `71fb2483f3..7c434b5758` that add work to every capture: #2644 (reading the `NotEnabled` trait per
158+
node), #2670 (publishing the captured keyboard band as a fact), #2693 (disclosing target
159+
re-activation on every capture-consuming command), and #2661 (the single geometry-normalization
160+
pass). This does not weaken the claims the corpora were measured for — one converged snapshot path,
161+
no presentation branching in the assembly, the RTT and cold-cold improvements above — but it is a
162+
real cost this head carries and it needs its own follow-up.
163+
164+
## Correction — the #2198 slice corpora are not parity baselines
165+
166+
The slice corpora published under `2198-slice-a-7616ba222d/`,
167+
`2198-slice-a-first-interaction-fixed-e729321dcc/`, and `2198-slice-b-rtt-318d510769/` were measured
168+
on branch heads (`7616ba222d`, `9189275dcb`, `e729321dcc`) that are **not ancestors** of `main`, and
169+
two of their first-interaction cells carry ten failed samples each (the ambiguous-anchor harness bug
170+
those slices themselves fixed). They are cited here as branch-lineage observations only. The pinned
171+
`71fb2483f3` corpus is the comparison this record rests on.
172+
173+
## Reproduction
174+
175+
```sh
176+
# one dedicated simulator, app installed, nothing else running a device
177+
pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \
178+
--mode local --state cold-cold,cold --samples 10 --skip-package-size --out .tmp/cold.json
179+
pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \
180+
--mode local --state first-interaction --samples 10 --skip-package-size --out .tmp/first.json
181+
pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \
182+
--mode local --state warm,relaunch --samples 20 --out .tmp/warm-relaunch.json
183+
pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \
184+
--mode proxy --rtt 0,20,80 --samples 20 --bandwidth-kbps unlimited --packet-loss 0 --skip-package-size \
185+
--out .tmp/proxy.json
186+
pnpm bench:ios-snapshot:evidence # schema and hash check against PUBLISHED_CORPORA
187+
```
188+
189+
Legs must not overlap: one subprocess-backed gate per host, otherwise the samples contend and the
190+
cell admission stops the run rather than reporting mixed timings.

‎scripts/ios-snapshot-benchmark/evidence.test.ts‎

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import { afterEach, test } from 'vitest';
66
import {
77
DEFAULT_EVIDENCE_DIR,
88
EVIDENCE_FIXTURE_PATH,
9+
PUBLISHED_CORPORA,
910
PUBLISHED_EVIDENCE,
1011
checkEvidenceCorpus,
1112
fetchEvidenceCommand,
@@ -80,13 +81,32 @@ test('the evidence README cites every published hash and the fetch recipe', () =
8081
assert.ok(readme.includes(fetchEvidenceCommand()));
8182
});
8283

83-
function oneOfThreePublishedFiles(): EvidenceFile {
84-
const [file, sha256] = Object.entries(PUBLISHED_EVIDENCE)[0]!;
84+
test('each published corpus is disjoint, named for its revision, and cited by tag and commit', () => {
85+
const readme = fs.readFileSync(path.join(DEFAULT_EVIDENCE_DIR, 'README.md'), 'utf8');
86+
const seen = new Set<string>();
87+
for (const corpus of PUBLISHED_CORPORA) {
88+
assert.match(corpus.tag, /^refs\/tags\/evidence\/ios-snapshot\/[0-9a-f]{9,}$/);
89+
assert.match(corpus.commit, /^[0-9a-f]{40}$/);
90+
assert.ok(readme.includes(corpus.tag), `README does not cite ${corpus.tag}`);
91+
assert.ok(readme.includes(corpus.commit), `README does not cite ${corpus.commit}`);
92+
for (const file of Object.keys(corpus.files)) {
93+
assert.ok(!seen.has(file), `${file} is published by two corpora`);
94+
seen.add(file);
95+
assert.ok(
96+
file.endsWith(`-${corpus.tag.split('/').pop()}.json`),
97+
`${file} is not named for ${corpus.tag}`,
98+
);
99+
}
100+
}
101+
});
102+
103+
function onePublishedFile(): EvidenceFile {
104+
const [file, sha256] = Object.entries(PUBLISHED_CORPORA[0]!.files)[0]!;
85105
return {
86106
file,
87107
sha256,
88108
published: 'match',
89-
revision: '71fb2483f30d90e615e949601c836aeebbf450c5',
109+
revision: PUBLISHED_CORPORA[0]!.revision,
90110
status: 'completed',
91111
cells: 2,
92112
errors: [],
@@ -95,15 +115,23 @@ function oneOfThreePublishedFiles(): EvidenceFile {
95115

96116
test('the default evidence directory must hold the complete published corpus, named files and all', () => {
97117
const dir = temporaryEvidenceDir({});
98-
assert.throws(
99-
() => checkEvidenceCorpus(dir, [oneOfThreePublishedFiles()], true),
100-
/is missing published evidence file\(s\): ios-snapshot-warm-relaunch-local-71fb2483f\.json, ios-snapshot-proxy-71fb2483f\.json/,
101-
);
118+
const present = onePublishedFile();
119+
let message = '';
120+
try {
121+
checkEvidenceCorpus(dir, [present], true);
122+
} catch (error) {
123+
message = error instanceof Error ? error.message : String(error);
124+
}
125+
assert.match(message, /is missing published evidence file\(s\)/);
126+
for (const file of Object.keys(PUBLISHED_EVIDENCE)) {
127+
if (file === present.file) continue;
128+
assert.ok(message.includes(file), `the error omits ${file}`);
129+
}
102130
});
103131

104132
test('an explicit --evidence-dir stays permissive: a partial corpus does not fail completeness', () => {
105133
const dir = temporaryEvidenceDir({});
106-
assert.doesNotThrow(() => checkEvidenceCorpus(dir, [oneOfThreePublishedFiles()], false));
134+
assert.doesNotThrow(() => checkEvidenceCorpus(dir, [onePublishedFile()], false));
107135
});
108136

109137
test('fetched evidence under the in-tree directory matches the published corpus', (context) => {

0 commit comments

Comments
 (0)