|
| 1 | +# iOS snapshot convergence final-head evidence |
| 2 | + |
| 3 | +- Issues: #2188 (gate), #2199 (measurement), #2189 (baseline corpus), #1571/#1626 (corpus lineage) |
| 4 | +- Observed: 2026-09-21, 19:03–22:52 CEST (legs sequential, one subprocess-backed gate per host) |
| 5 | +- Revision: `7c434b5758` (`docs/ios-snapshot-final-evidence`, clean tree at every leg; `revision.dirty: false`) |
| 6 | +- Target: `bench-2188-final` iPhone 17 Pro Simulator, iOS 26.2, UDID `5FEADD02-98E4-4F01-861C-07003C1A3291` |
| 7 | +- Host: MacBook Pro `Mac16,8`, Apple M4 Pro, 12 cores, Xcode 26.2 (17C52), Node v26.8.1 |
| 8 | +- App: `Agent Device Tester` (`com.callstack.agentdevicelab`) from the CI artifact |
| 9 | + `fingerprint.d6b4c1f5e022e54074df584e332d1e0f100c7a0f.ios` (run `35530285864`), repacked with the |
| 10 | + measured head's own JavaScript bundle |
| 11 | +- Corpora: tag `evidence/ios-snapshot/7c434b575`, evidence commit `96d4951c19`, four files listed |
| 12 | + with their SHA-256 digests in [`scripts/ios-snapshot-benchmark/evidence/README.md`](../../scripts/ios-snapshot-benchmark/evidence/README.md) |
| 13 | +- Baseline: tag `evidence/ios-snapshot/71fb2483f`, evidence commit `2d4baf461aa`, measured at |
| 14 | + `71fb2483f3` on `bench-golden-v2` (iOS 27.0) |
| 15 | + |
| 16 | +## What this closes |
| 17 | + |
| 18 | +#2188 was gated on #2199's final exact-head evidence sweep: the package-size delta against #2189, the |
| 19 | +conformance/fuzz/property corpus, provider contracts, the Simulator and proxy legs, the no-regrowth |
| 20 | +enforcement, and a release record at a named head. All of it now exists at `7c434b5758`. |
| 21 | + |
| 22 | +The proxy leg was the last blocker, and it was blocked by the harness, not by the product. The |
| 23 | +anchor admission the harness gained at `71fb2483f3` reads the tree exactly once after `apps.open`. |
| 24 | +On this target the first tree lands later than that read: the snapshot taken immediately after an |
| 25 | +`open --relaunch --launch-url` returns the single `Application` node, while a read 1.5 s later |
| 26 | +returns 31 nodes with the `Catalog` anchor. Every deep-linked screen the leg reached stopped with |
| 27 | +`fixture-anchor`. `45e4c594a1..7c434b5758` puts a bounded wait into that untimed setup admission |
| 28 | +(`FIXTURE_ANCHOR_ADMISSION_BUDGET_MS`, 30 s, 500 ms polling) and keeps the same typed stop when a |
| 29 | +fixture never exposes its anchor. No `src/`, `packages/`, app, or CLI runtime file differs between |
| 30 | +those two commits; the measured binary is the measured head's own `pnpm build` output. |
| 31 | + |
| 32 | +## Deviations from the baseline corpus |
| 33 | + |
| 34 | +Stated in full, because they bound what these numbers can prove. |
| 35 | + |
| 36 | +| Deviation | This corpus | Baseline corpus | |
| 37 | +| --- | --- | --- | |
| 38 | +| Simulator and runtime | `bench-2188-final`, iOS 26.2 | `bench-golden-v2`, iOS 27.0 | |
| 39 | +| Why | `bench-golden-v2`'s iOS 27.0 runtime was not installed on this host at measurement time | — | |
| 40 | +| Host load (1-min average, sampled per leg) | warm/relaunch 4.2–31.2, proxy 6.4–30.3, cold 5.3–96.7 | quiet | |
| 41 | +| Fixture app trees | catalog 31 nodes, nested-scroll 23, checkout 25, alert 26, Settings 18, inert 4 | catalog 35, nested-scroll 25, checkout 29, alert 28, Settings 18, inert 6 | |
| 42 | +| Package-size span | `71fb2483f3..7c434b5758`: 196 commits of unrelated product work | — | |
| 43 | + |
| 44 | +Because the fixture trees differ, response sizes are not comparable cell for cell, and any timing |
| 45 | +delta on a screen whose tree changed is only partly attributable to the tooling. Timings are bounded |
| 46 | +observations under uncontrolled host load, not a general performance guarantee. |
| 47 | + |
| 48 | +## Result — cold and cold-cold (10 samples per cell, fresh CLI process) |
| 49 | + |
| 50 | +| State | Screen | Wall median | Wall median (base) | Daemon median | Daemon median (base) | |
| 51 | +| --- | --- | ---: | ---: | ---: | ---: | |
| 52 | +| cold-cold | quiet | 7,159 | 17,544 | 5,709 | 15,161 | |
| 53 | +| cold-cold | list | 7,580 | 19,170 | 5,881 | 17,191 | |
| 54 | +| cold-cold | nested-scroll | 8,091 | 19,407 | 6,374 | 17,773 | |
| 55 | +| cold-cold | alert | 7,461 | 19,448 | 5,689 | 17,436 | |
| 56 | +| cold-cold | system-surface | 6,553 | 20,235 | 4,961 | 17,974 | |
| 57 | +| cold-cold | xctest-stress | 7,273 | 20,811 | 5,761 | 18,506 | |
| 58 | +| cold | quiet | 5,743 | 6,643 | 4,409 | 5,886 | |
| 59 | +| cold | list | 5,847 | 6,851 | 4,420 | 5,897 | |
| 60 | +| cold | nested-scroll | 6,058 | 6,713 | 4,619 | 5,884 | |
| 61 | +| cold | alert | 5,706 | 6,674 | 4,310 | 5,868 | |
| 62 | +| cold | system-surface | 5,028 | 6,819 | 3,662 | 5,957 | |
| 63 | +| cold | xctest-stress | 5,679 | 6,714 | 4,323 | 5,874 | |
| 64 | + |
| 65 | +Milliseconds. `cold-cold` shuts the Simulator down, clears benchmark-owned derived data, and boots it |
| 66 | +again before every sample; `cold` stops the daemon and terminates the app. Cold-cold wall medians |
| 67 | +fall by 57–68% and daemon medians by 62–72% against the baseline, measured while this host carried |
| 68 | +substantially more load than the baseline's. Cold (booted Simulator, cold runner) improves by 9–26%. |
| 69 | + |
| 70 | +## Result — warm, relaunch, and first interaction |
| 71 | + |
| 72 | +Warm and relaunch carry 20 samples per cell; first interaction carries 10 and has no baseline-corpus |
| 73 | +counterpart, so it is reported on its own. |
| 74 | + |
| 75 | +| Screen | Warm wall | Warm wall (base) | Warm daemon | Warm daemon (base) | Relaunch wall | Relaunch wall (base) | First interaction wall | |
| 76 | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| 77 | +| quiet | 169 | 154 | 89 | 50 | 3,876 | 3,645 | 928 | |
| 78 | +| list | 291 | 312 | 202 | 209 | 3,888 | 4,378 | 1,436 | |
| 79 | +| nested-scroll | 250 | 187 | 167 | 83 | 3,872 | 4,313 | 741 | |
| 80 | +| alert | 224 | 207 | 139 | 104 | 3,896 | 4,370 | 1,005 | |
| 81 | +| system-surface | 208 | 210 | 130 | 105 | 3,576 | 4,917 | 1,119 | |
| 82 | +| xctest-stress | 213 | 208 | 128 | 105 | 3,868 | 4,260 | 1,302 | |
| 83 | + |
| 84 | +Milliseconds. Relaunch is at or better than the baseline on five of six screens. Warm wall is within |
| 85 | +±10% of the baseline on five of six screens; warm daemon duration is uniformly higher, which is the |
| 86 | +finding below. |
| 87 | + |
| 88 | +## Result — proxy transport (20 samples per screen at each added RTT) |
| 89 | + |
| 90 | +The headline of the convergence work: a warm snapshot crossed the tunnel with far fewer round trips |
| 91 | +than at the baseline, so added latency costs far less. |
| 92 | + |
| 93 | +| Screen | Transport | 0 ms | 20 ms | 80 ms | 0 ms (base) | 20 ms (base) | 80 ms (base) | |
| 94 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| 95 | +| quiet | fresh CLI | 184 | 293 | 556 | 233 | 1,057 | 1,801 | |
| 96 | +| quiet | persistent client | 103 | 158 | 282 | 273 | 562 | 806 | |
| 97 | +| list | fresh CLI | 304 | 415 | 691 | 431 | 1,221 | 1,328 | |
| 98 | +| list | persistent client | 214 | 280 | 408 | 254 | 719 | 729 | |
| 99 | +| nested-scroll | fresh CLI | 260 | 377 | 643 | 188 | 1,116 | 1,108 | |
| 100 | +| nested-scroll | persistent client | 179 | 240 | 381 | 138 | 590 | 616 | |
| 101 | +| alert | fresh CLI | 241 | 343 | 615 | 210 | 1,113 | 1,362 | |
| 102 | +| alert | persistent client | 148 | 203 | 351 | 108 | 622 | 637 | |
| 103 | +| system-surface | fresh CLI | 217 | 319 | 595 | 209 | 1,119 | 1,145 | |
| 104 | +| system-surface | persistent client | 128 | 183 | 330 | 105 | 537 | 638 | |
| 105 | +| xctest-stress | fresh CLI | 229 | 340 | 562 | 225 | 1,136 | 1,395 | |
| 106 | +| xctest-stress | persistent client | 136 | 197 | 314 | 127 | 612 | 619 | |
| 107 | + |
| 108 | +Milliseconds, wall-clock medians. Going from 0 to 80 ms of added RTT costs 178–387 ms per snapshot |
| 109 | +here, against 475–1,568 ms at the baseline. On the quiet fixture over the fresh CLI transport that |
| 110 | +difference is about twenty RTT-bound exchanges per snapshot at the baseline against about five here; |
| 111 | +over the persistent client it is about seven against about two. Every cell completed with zero failed |
| 112 | +samples at every RTT, so the retry/timeout path is not what improved. |
| 113 | + |
| 114 | +## Result — package size against #2189 |
| 115 | + |
| 116 | +| Measurement | `7c434b5758` | `71fb2483f3` | Delta | |
| 117 | +| --- | ---: | ---: | ---: | |
| 118 | +| Packed tarball | 1,409,306 B | 982,960 B | +43.4% | |
| 119 | +| Clean-installed package | 4,723,895 B / 543 files | 3,347,347 B / 436 files | +41.1% / +107 | |
| 120 | +| Bundled JavaScript | 3,775,024 B raw, 1,258,950 B gzip, 410 files | 2,480,947 B raw, 835,279 B gzip, 330 files | +52.2% raw, +50.7% gzip | |
| 121 | + |
| 122 | +The span covers 196 commits of unrelated product work, so this delta cannot be attributed to the |
| 123 | +snapshot convergence alone; it is recorded as the size at the measured head, and the leg reproduces |
| 124 | +to the byte across independent runs (the same `1,409,306` tarball twice). |
| 125 | + |
| 126 | +## Result — deterministic corpus and no-regrowth enforcement |
| 127 | + |
| 128 | +| Gate | Command | Result | |
| 129 | +| --- | --- | --- | |
| 130 | +| Swift/TypeScript presentation differential | `pnpm test:ios-snapshot-differential` | pass (`swift test --package-path apple/snapshot-presentation`, plus 3 differential tests) | |
| 131 | +| Engine conformance, properties, transitions, tree | `vitest run packages/capture-kit/src/ios-snapshot-engine` | pass (10 files, 67 tests) | |
| 132 | +| Fuzz corpus replay | `vitest run scripts/fuzz/corpus-replay.test.ts` | pass (11 cases) | |
| 133 | +| Fuzz worker lane | `pnpm test:fuzz-worker` | pass (11 cases) | |
| 134 | +| Provider contracts | `pnpm test:integration:provider` | pass (66 files, 212 tests) | |
| 135 | +| No-regrowth rules R72/R73/R74 | `pnpm check:layering` | pass (`ios-snapshot-engine-ownership`, `provider-snapshot-presentation-ownership`, `snapshot-assembly-presentation-neutrality`, wired in `scripts/layering/check.ts`) | |
| 136 | +| Benchmark harness unit suite | `vitest run scripts/ios-snapshot-benchmark` | pass (17 files, 50 tests) | |
| 137 | +| Deep-button controls | inside every corpus (`deepButtonEvidence`) | the shallow-rule control exits 1 with "changed descendant was omitted by shallow observation"; the safe control exits 0 | |
| 138 | +| Affected gate | `pnpm check:affected --run` | see the PR; GitHub stays authoritative for provider integration, coverage, native builds, and device lanes | |
| 139 | + |
| 140 | +## Finding — warm snapshot daemon cost is about double the baseline |
| 141 | + |
| 142 | +Recorded here rather than hidden, and not explained by target, runtime, app, or load. |
| 143 | + |
| 144 | +An off-corpus controlled diagnostic: same Simulator, same installed app, same ambient load |
| 145 | +(1-minute average ~21), back-to-back runs of ten warm `interactiveOnly` snapshots of the smallest |
| 146 | +fixture (4 nodes, ~3.4 KB), reading the daemon's own per-step duration. |
| 147 | + |
| 148 | +| CLI under test | Durations (ms) | Median | |
| 149 | +| --- | --- | ---: | |
| 150 | +| `71fb2483f3` (baseline corpus head) | 68, 47, 46, 44, 43, 43, 43, 45, 47, 46 | 45.5 | |
| 151 | +| `7c434b5758` (this head), round 1 | 97, 86, 84, 90, 84, 86, 86, 92, 83, 82 | 86 | |
| 152 | +| `7c434b5758` (this head), round 2 | 98, 133, 96, 96, 88, 90, 87, 94, 107, 103 | 96 | |
| 153 | + |
| 154 | +The published warm cells agree with the diagnostic (89–202 ms against the baseline's 50–209 ms), and |
| 155 | +a repeat at a 1-minute load average of 3.8 still measured 86 ms, so contention is not the cause. The |
| 156 | +shape is a roughly fixed ~40 ms per snapshot rather than work proportional to tree size. Candidates |
| 157 | +in `71fb2483f3..7c434b5758` that add work to every capture: #2644 (reading the `NotEnabled` trait per |
| 158 | +node), #2670 (publishing the captured keyboard band as a fact), #2693 (disclosing target |
| 159 | +re-activation on every capture-consuming command), and #2661 (the single geometry-normalization |
| 160 | +pass). This does not weaken the claims the corpora were measured for — one converged snapshot path, |
| 161 | +no presentation branching in the assembly, the RTT and cold-cold improvements above — but it is a |
| 162 | +real cost this head carries and it needs its own follow-up. |
| 163 | + |
| 164 | +## Correction — the #2198 slice corpora are not parity baselines |
| 165 | + |
| 166 | +The slice corpora published under `2198-slice-a-7616ba222d/`, |
| 167 | +`2198-slice-a-first-interaction-fixed-e729321dcc/`, and `2198-slice-b-rtt-318d510769/` were measured |
| 168 | +on branch heads (`7616ba222d`, `9189275dcb`, `e729321dcc`) that are **not ancestors** of `main`, and |
| 169 | +two of their first-interaction cells carry ten failed samples each (the ambiguous-anchor harness bug |
| 170 | +those slices themselves fixed). They are cited here as branch-lineage observations only. The pinned |
| 171 | +`71fb2483f3` corpus is the comparison this record rests on. |
| 172 | + |
| 173 | +## Reproduction |
| 174 | + |
| 175 | +```sh |
| 176 | +# one dedicated simulator, app installed, nothing else running a device |
| 177 | +pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \ |
| 178 | + --mode local --state cold-cold,cold --samples 10 --skip-package-size --out .tmp/cold.json |
| 179 | +pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \ |
| 180 | + --mode local --state first-interaction --samples 10 --skip-package-size --out .tmp/first.json |
| 181 | +pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \ |
| 182 | + --mode local --state warm,relaunch --samples 20 --out .tmp/warm-relaunch.json |
| 183 | +pnpm bench:ios-snapshot -- --udid <UDID> --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \ |
| 184 | + --mode proxy --rtt 0,20,80 --samples 20 --bandwidth-kbps unlimited --packet-loss 0 --skip-package-size \ |
| 185 | + --out .tmp/proxy.json |
| 186 | +pnpm bench:ios-snapshot:evidence # schema and hash check against PUBLISHED_CORPORA |
| 187 | +``` |
| 188 | + |
| 189 | +Legs must not overlap: one subprocess-backed gate per host, otherwise the samples contend and the |
| 190 | +cell admission stops the run rather than reporting mixed timings. |
0 commit comments