You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f4cd502
Browse filesBrowse the repository at this point in the historyBrowse files
test(ios): drive the guest Simulator AX bridge directly from Node
Replace the idb companion + Python reader in the #2192 spike with a Node client
for idb v1.5.2's in-Simulator SimulatorFrameworkBridge: one private guest per
session spawned through simctl, 4-byte length-prefixed JSON over a UNIX socket,
single-fetch traversal with automation mode asserted per request, nested trees
flattened to parent-linked raw nodes with XCTest type names, and typed
crash/timeout/cancel/stale-generation failures.
The targeted harness now observes app readiness with a throwaway probe instead
of admitting on pid presence, relaunches the app per bootstrap sample, records
host load per sample, and runs recovery probes through the adapter. Hard tiers
follow the corrected #2192 contract (warm 300/500 ms, relaunch 500 ms); the
former 75/150 ms and 250 ms values are reported as stretch findings. Preboot
preference edits are optional and unused by the guest path.
The prototype's targeted artifact is preserved under a -python-prototype name;
its bootstrap and recovery samples measured the packaging, not the mechanism.
- Immutable broad raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-01-final.json.gz` (original NO-GO; interpretation superseded to stretch-only; host client persistent-in-repository-reader)
9
+
- Superseded targeted raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted-python-prototype.json.gz` (persistent-in-repository-reader (idb_companion + Python idb client); its bootstrap and recovery samples raced app readiness and shared one wedged companion, so they measured the prototype packaging, not the mechanism)
10
+
- Narrow targeted raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted.json.gz` (host client node-direct-socket)
11
+
- Host at generation: load average 13.75 on 12 cores
10
12
11
-
The broad run is preserved unchanged. Its old NO-GO was caused by readiness-inclusive first-look and stretch thresholds; this report evaluates the corrected hard contract.
13
+
The broad run is preserved unchanged. Its old NO-GO was caused by readiness-inclusive first-look and stretch thresholds; this report evaluates the corrected hard contract. Warm and relaunch cells come from the broad run, whose host client was the idb companion plus a Python reader; the in-Simulator reader and the read it performs are the same mechanism the Node-direct targeted evidence uses, and the host client only adds latency, so those cells bound the mechanism from above.
- Traversal: describe with snapshotTree=true (one XCTest snapshot fetch per read) and automationMode=true asserted per request; no idb_companion, gRPC, or Python client.
12
20
13
21
## Hard gates
14
22
15
23
| Gate | Status | Target | Evidence |
16
24
|---|---|---|---|
17
25
| warm |**PASS**| p50 <300 ms and p95 <500 ms per screen | 6/6 warm screen cells passed; quiet p50/p95=8.6 ms/9.3 ms ready=20/20; list p50/p95=118.2 ms/120.9 ms ready=20/20; nested-scroll p50/p95=15.1 ms/15.8 ms ready=20/20; alert p50/p95=41.6 ms/43.3 ms ready=20/20; system-surface p50/p95=37.2 ms/39.6 ms ready=20/20; xctest-stress p50/p95=39.6 ms/41.1 ms ready=20/20 |
18
26
| relaunch |**PASS**| p95 <500 ms per screen after observed new-generation app readiness | 6/6 relaunch screen cells passed; quiet p50/p95=8.9 ms/9.6 ms ready=20/20; list p50/p95=119.2 ms/121.1 ms ready=20/20; nested-scroll p50/p95=15.4 ms/16.5 ms ready=20/20; alert p50/p95=41.1 ms/42.7 ms ready=20/20; system-surface p50/p95=37.3 ms/39.5 ms ready=20/20; xctest-stress p50/p95=40.4 ms/42.6 ms ready=20/20 |
19
-
| nonresidentBootstrap |**FAIL**| nonresident companion + reader bootstrap and first usable tree p95 <2,000 ms |1/5 usable trees; p95=5768.8 ms; timer covered adapter acquireBatch only after app readiness, with no xcodebuild, XCTest, or agent-device runner in the timed path |
20
-
| liveRecovery |**FAIL**| live crash, timeout, cancellation, and honest target-generation handling |0/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response |
21
-
|hierarchyResidue|**PASS**|missing hierarchy represented as typed provider-pruned depth residue |provider-pruned/depth observed; traversal depth is not treated as complete|
27
+
| nonresidentBootstrap |**PASS**| nonresident companion + reader bootstrap and first usable tree p95 <2,000 ms |5/5 usable trees; p95=1136.2 ms; the timer covered guest spawn, socket connect, and the first tree after a throwaway probe observed the relaunched app's readiness (readiness p95=1333.2 ms), with no resident bridge, xcodebuild, XCTest, or agent-device runner in the timed path |
28
+
| liveRecovery |**PASS**| live crash, timeout, cancellation, and honest target-generation handling |4/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response |
29
+
|hierarchy|**PASS**|structural hierarchy acquired with typed truncation, or its absence typed as residue |nested tree with traversal depth 29 in 5/5 samples; truncated=false|
22
30
23
31
## Readiness boundary and candidate-owned latency
24
32
@@ -60,32 +68,32 @@ Cold and cold-cold first-look measurements remain visible for diagnosis, but are
60
68
61
69
## Nonresident bootstrap
62
70
63
-
-1/5 usable trees; p95=5768.8 ms; timer covered adapter acquireBatch only after app readiness, with no xcodebuild, XCTest, or agent-device runner in the timed path.
64
-
- The timed boundary begins with a nonresident adapter and ends at the first usable guest tree; Simulator/app readiness was established before the timer.
71
+
-5/5 usable trees; p95=1136.2 ms; the timer covered guest spawn, socket connect, and the first tree after a throwaway probe observed the relaunched app's readiness (readiness p95=1333.2 ms), with no resident bridge, xcodebuild, XCTest, or agent-device runner in the timed path.
72
+
- The timed boundary begins with no resident bridge and ends at the first usable guest tree. Before each timer the fixture app was relaunched and a throwaway probe bridge polled until the new generation answered with a tree (readiness), then exited.
65
73
66
-
| Sample | Duration ms | Usable tree | Failure | Nodes | Generation |
@@ -95,6 +103,7 @@ Cold and cold-cold first-look measurements remain visible for diagnosis, but are
95
103
- Original broad-run finding: guest-simulator-framework-bridge relaunch first look missed the 250 ms target.
96
104
- Cold and cold-cold first-look measurements include Simulator, app, daemon, and runner readiness costs; they are diagnostics, not candidate-owned hard gates.
97
105
- The former warm 75/150 ms and relaunch 250 ms thresholds are stretch findings under the corrected contract.
106
+
- Nonresident bootstrap samples were taken on a host with 1-minute load average 13.75 on 12 cores; per-sample load is recorded with each sample.
This bounded harness supplies the decision evidence for [#2192](https://github.com/callstack/agent-device/issues/2192). It compares the official idb `SimulatorFrameworkBridge` guest mechanism with the #2189 XCTest control baseline behind one acquisition adapter. It does not select a production backend or change daemon, runner, open, relaunch, proxy, interaction, or public CLI behavior.
3
+
This bounded harness supplies the decision evidence for [#2192](https://github.com/callstack/agent-device/issues/2192). It compares an in-Simulator accessibility reader with the #2189 XCTest control baseline behind one acquisition adapter. It does not select a production backend or change daemon, runner, open, relaunch, proxy, interaction, or public CLI behavior, and nothing in it ships in the npm package.
4
+
5
+
## Mechanism under test
6
+
7
+
The guest candidate is idb v1.5.2's `Resources/SimulatorFrameworkBridge`: a 196 KB iOS-Simulator executable that reads the XCTest-shaped element tree (`XC_kAXXCAttribute*`) inside the Simulator and serves it over a UNIX socket. The spike drives it **directly from Node**:
8
+
9
+
-`xcrun simctl spawn <udid> SimulatorFrameworkBridge accessibility serve <socket> --idle-timeout 300 --exit-on-disconnect true` starts one private guest per session in the Simulator's launchd domain;
10
+
- the host connects to the socket and exchanges 4-byte big-endian length-prefixed JSON frames;
11
+
- each read is one `describe` with `snapshotTree=true` (one XCTest snapshot fetch, one Mach round trip) and `automationMode=true`, so the target exposes its accessibility server without preboot preference edits;
12
+
- a known target generation reads by pid; otherwise the guest resolves the foreground app in-guest through RunningBoard.
13
+
14
+
No `idb_companion`, gRPC, or Python client is involved. The earlier prototype packaging (companion + Python reader) is retained only as the superseded targeted artifact.
4
15
5
16
Build the repository first:
6
17
@@ -9,7 +20,9 @@ pnpm install --frozen-lockfile
9
20
pnpm build
10
21
```
11
22
12
-
Use a newly created, task-owned iOS Simulator. The guest candidate uses the arm64 [idb release](https://github.com/facebook/idb/releases/tag/v1.5.2) outside this repository:
23
+
Obtain the guest executable from the official arm64 [idb v1.5.2 release](https://github.com/facebook/idb/releases/tag/v1.5.2) (`idb-companion.macos-arm64.tar.gz`, SHA-256 `f17b718a513931705542a7fbfa9cfc11895ee191562c9ffd2343cf7f8254bc08`; the extracted `Resources/SimulatorFrameworkBridge` has SHA-256 `3545621d2dc98de32879ebac55e8b0c33dc8eb7cc2bfbc2d0d2d21a002c8de58`). Use a task-owned iOS Simulator with the test app installed.
The default candidate set, state set, screen set, and sample minimums come from the #2189 benchmark definitions. Each request carries an optional expected target generation and fixed request, response, node-count, traversal-depth, CPU, memory, and duration bounds. The guest adapter uses a newline-delimited reader, keeping one idb gRPC client and one `axbridge-persistent` reader alive across the run. Guest reads request idb's flat raw element form, preserving provider facts without importing visibility, hittability, scope, depth, or semantic compaction. Every sample keeps resource metrics and target status; each cell keeps one raw-tree exemplar with viewport, lineage, truncation, residue, and bounded diagnostics, plus separate prototype presentation measurements.
38
+
The default candidate set, state set, screen set, and sample minimums come from the #2189 benchmark definitions. Each request carries an optional expected target generation and fixed request, response, node-count, traversal-depth, CPU, memory, and duration bounds. Guest reads keep the nested view hierarchy as parent-linked raw nodes with XCTest type names, labels, values, identifiers, and frames; they do not import visibility, hittability, scope, depth, or semantic compaction. Every sample keeps resource metrics and target status; each cell keeps one raw-tree exemplar with viewport, lineage, truncation, residue, and bounded diagnostics, plus separate prototype presentation measurements.
39
+
40
+
Hard tiers follow the maintainer-corrected #2192 contract: warm p50 < 300 ms and p95 < 500 ms per screen, relaunch p95 < 500 ms after observed app readiness. The former 75/150 ms and 250 ms values are reported as stretch findings and never decide GO/NO-GO.
29
41
30
-
`--apply-preferences`is the only way the experiment edits Simulator preference plists. The Simulator must be shutdown; the harness records exact plist hashes and targeted key changes, then restores the original bytes before reporting. The keys are not production defaults.
42
+
`--apply-preferences`optionally runs the task-owned preboot AX preference experiment. The Simulator must be shutdown; the harness records exact plist hashes and targeted key changes, then restores the original bytes before reporting. The keys are not production defaults and the guest path does not need them.
31
43
32
-
The harness fails closed. It reports `NO-GO` when the guest candidate is unsupported, unavailable, unreadable, stale, over a bound, below the sample minimum, or when crash/timeout/cancellation recovery is not typed and recovered. XCTest is a control result and cannot turn a passing guest corpus into a failure. It stops before reporting timings if the fixture app cannot be prepared deterministically. The adjacent gzipped JSON and readable Markdown report are the decision artifact; no production route should be implemented from a `NO-GO` run.
44
+
## Targeted evidence
33
45
34
-
The checked-in broad corpus predates the corrected hard-latency contract. Reproduce only the
35
-
missing live bootstrap and lifecycle evidence with:
46
+
The broad corpus predates the corrected hard-latency contract. The live nonresident-bootstrap and lifecycle evidence is produced with:
This preserves the broad raw artifact and writes a narrow raw artifact plus the superseding
47
-
corrected report. Each nonresident bootstrap sample re-establishes a booted Simulator and ready app
48
-
before its timer, so helper teardown contention and Simulator/app readiness are outside the measured
49
-
candidate boundary. Missing provider generation is emitted as typed residue; the reader never echoes
50
-
an expected generation it did not observe.
54
+
For each of five bootstrap samples the fixture app is relaunched, a throwaway probe bridge polls until the new app generation answers with a tree (readiness is observed, never assumed from a pid), the probe exits, and only then a fresh guest is spawned and timed to its first usable tree. Host load is recorded per sample. Recovery probes exercise process crash, timeout, cancellation, and a dead target generation through the same adapter and require a typed failure plus a usable recovered read. The run preserves the broad raw artifact, writes the narrow raw artifact, and regenerates the corrected report.
55
+
56
+
The harness fails closed. It reports `NO-GO` when the guest candidate is unsupported, unavailable, unreadable, stale, over a bound, below the sample minimum, or when crash/timeout/cancellation recovery is not typed and recovered. XCTest is a control result and cannot turn a passing guest corpus into a failure.
0 commit comments