Skip to content

Commit f4cd502

Browse files
committed
test(ios): drive the guest Simulator AX bridge directly from Node
Replace the idb companion + Python reader in the #2192 spike with a Node client for idb v1.5.2's in-Simulator SimulatorFrameworkBridge: one private guest per session spawned through simctl, 4-byte length-prefixed JSON over a UNIX socket, single-fetch traversal with automation mode asserted per request, nested trees flattened to parent-linked raw nodes with XCTest type names, and typed crash/timeout/cancel/stale-generation failures. The targeted harness now observes app readiness with a throwaway probe instead of admitting on pid presence, relaunches the app per bootstrap sample, records host load per sample, and runs recovery probes through the adapter. Hard tiers follow the corrected #2192 contract (warm 300/500 ms, relaunch 500 ms); the former 75/150 ms and 250 ms values are reported as stretch findings. Preboot preference edits are optional and unused by the guest path. The prototype's targeted artifact is preserved under a -python-prototype name; its bootstrap and recovery samples measured the packaging, not the mechanism.
1 parent b6a1f4d commit f4cd502

27 files changed

Lines changed: 1518 additions & 1082 deletions
Binary file not shown.

‎docs/evidence/ios-simulator-ax-bridge-2026-09-02-corrected.md‎

Lines changed: 36 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,32 @@
11
# iOS Simulator AX bridge corrected evidence
22

3-
- Decision: **NO-GO**
3+
- Decision: **GO**
44
- Interpretation: **maintainer-corrected**
5-
- Revision: eac2c7f409f4148bbeb1af87a55ad74eef54e8fc (codex/2192-guest-bridge-evidence)
6-
- Target: bench-golden-v2 (7E76ECA9-D40C-4833-A711-F870F8CE9363, com.apple.CoreSimulator.SimRuntime.iOS-27-0)
7-
- Generated: 2026-09-02T15:37:43.300Z
8-
- Immutable broad raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-01-final.json.gz` (original NO-GO; interpretation superseded to stretch-only)
9-
- Narrow targeted raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted.json.gz`
5+
- Revision: 999920fa55098f11eb5ba1f9d39f9cb3cec208e3 (review-2237)
6+
- Target: ad-2237-axbridge (8CDB4DF1-3A3E-4FB1-AF89-B3D3A17647D5, com.apple.CoreSimulator.SimRuntime.iOS-26-2)
7+
- Generated: 2026-09-03T06:04:19.830Z
8+
- Immutable broad raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-01-final.json.gz` (original NO-GO; interpretation superseded to stretch-only; host client persistent-in-repository-reader)
9+
- Superseded targeted raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted-python-prototype.json.gz` (persistent-in-repository-reader (idb_companion + Python idb client); its bootstrap and recovery samples raced app readiness and shared one wedged companion, so they measured the prototype packaging, not the mechanism)
10+
- Narrow targeted raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted.json.gz` (host client node-direct-socket)
11+
- Host at generation: load average 13.75 on 12 cores
1012

11-
The broad run is preserved unchanged. Its old NO-GO was caused by readiness-inclusive first-look and stretch thresholds; this report evaluates the corrected hard contract.
13+
The broad run is preserved unchanged. Its old NO-GO was caused by readiness-inclusive first-look and stretch thresholds; this report evaluates the corrected hard contract. Warm and relaunch cells come from the broad run, whose host client was the idb companion plus a Python reader; the in-Simulator reader and the read it performs are the same mechanism the Node-direct targeted evidence uses, and the host client only adds latency, so those cells bound the mechanism from above.
14+
15+
## Evaluated guest mechanism
16+
17+
- Guest reader: idb v1.5.2 `Resources/SimulatorFrameworkBridge` (SHA-256 `3545621d2dc98de32879ebac55e8b0c33dc8eb7cc2bfbc2d0d2d21a002c8de58`) from `idb-companion.macos-arm64.tar.gz` (SHA-256 `f17b718a513931705542a7fbfa9cfc11895ee191562c9ffd2343cf7f8254bc08`).
18+
- Transport: xcrun simctl spawn <udid> SimulatorFrameworkBridge accessibility serve <socket> --idle-timeout 300 --exit-on-disconnect true; UNIX socket frames are a 4-byte big-endian length + JSON.
19+
- Traversal: describe with snapshotTree=true (one XCTest snapshot fetch per read) and automationMode=true asserted per request; no idb_companion, gRPC, or Python client.
1220

1321
## Hard gates
1422

1523
| Gate | Status | Target | Evidence |
1624
|---|---|---|---|
1725
| warm | **PASS** | p50 <300 ms and p95 <500 ms per screen | 6/6 warm screen cells passed; quiet p50/p95=8.6 ms/9.3 ms ready=20/20; list p50/p95=118.2 ms/120.9 ms ready=20/20; nested-scroll p50/p95=15.1 ms/15.8 ms ready=20/20; alert p50/p95=41.6 ms/43.3 ms ready=20/20; system-surface p50/p95=37.2 ms/39.6 ms ready=20/20; xctest-stress p50/p95=39.6 ms/41.1 ms ready=20/20 |
1826
| relaunch | **PASS** | p95 <500 ms per screen after observed new-generation app readiness | 6/6 relaunch screen cells passed; quiet p50/p95=8.9 ms/9.6 ms ready=20/20; list p50/p95=119.2 ms/121.1 ms ready=20/20; nested-scroll p50/p95=15.4 ms/16.5 ms ready=20/20; alert p50/p95=41.1 ms/42.7 ms ready=20/20; system-surface p50/p95=37.3 ms/39.5 ms ready=20/20; xctest-stress p50/p95=40.4 ms/42.6 ms ready=20/20 |
19-
| nonresidentBootstrap | **FAIL** | nonresident companion + reader bootstrap and first usable tree p95 <2,000 ms | 1/5 usable trees; p95=5768.8 ms; timer covered adapter acquireBatch only after app readiness, with no xcodebuild, XCTest, or agent-device runner in the timed path |
20-
| liveRecovery | **FAIL** | live crash, timeout, cancellation, and honest target-generation handling | 0/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response |
21-
| hierarchyResidue | **PASS** | missing hierarchy represented as typed provider-pruned depth residue | provider-pruned/depth observed; traversal depth is not treated as complete |
27+
| nonresidentBootstrap | **PASS** | nonresident companion + reader bootstrap and first usable tree p95 <2,000 ms | 5/5 usable trees; p95=1136.2 ms; the timer covered guest spawn, socket connect, and the first tree after a throwaway probe observed the relaunched app's readiness (readiness p95=1333.2 ms), with no resident bridge, xcodebuild, XCTest, or agent-device runner in the timed path |
28+
| liveRecovery | **PASS** | live crash, timeout, cancellation, and honest target-generation handling | 4/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response |
29+
| hierarchy | **PASS** | structural hierarchy acquired with typed truncation, or its absence typed as residue | nested tree with traversal depth 29 in 5/5 samples; truncated=false |
2230

2331
## Readiness boundary and candidate-owned latency
2432

@@ -60,32 +68,32 @@ Cold and cold-cold first-look measurements remain visible for diagnosis, but are
6068

6169
## Nonresident bootstrap
6270

63-
- 1/5 usable trees; p95=5768.8 ms; timer covered adapter acquireBatch only after app readiness, with no xcodebuild, XCTest, or agent-device runner in the timed path.
64-
- The timed boundary begins with a nonresident adapter and ends at the first usable guest tree; Simulator/app readiness was established before the timer.
71+
- 5/5 usable trees; p95=1136.2 ms; the timer covered guest spawn, socket connect, and the first tree after a throwaway probe observed the relaunched app's readiness (readiness p95=1333.2 ms), with no resident bridge, xcodebuild, XCTest, or agent-device runner in the timed path.
72+
- The timed boundary begins with no resident bridge and ends at the first usable guest tree. Before each timer the fixture app was relaunched and a throwaway probe bridge polled until the new generation answered with a tree (readiness), then exited.
6573

66-
| Sample | Duration ms | Usable tree | Failure | Nodes | Generation |
67-
|---:|---:|---|---|---:|---|
68-
| 1 | 1990.6 | true | none/none | 159 | – |
69-
| 2 | 5718.8 | false | timeout/batch-duration-limit | 0 | – |
70-
| 3 | 5702.4 | false | timeout/batch-duration-limit | 0 | – |
71-
| 4 | 5740.5 | false | timeout/batch-duration-limit | 0 | – |
72-
| 5 | 5768.8 | false | timeout/batch-duration-limit | 0 | – |
74+
| Sample | Duration ms | Usable tree | Failure | Nodes | Depth | Generation | Readiness ms | Readiness attempts | Host load |
75+
|---:|---:|---|---|---:|---:|---|---:|---:|---:|
76+
| 1 | 1080.8 | true | none/none | 155 | 29 | pid:68714 | 1333 | 1 | 11.66 |
77+
| 2 | 1136.2 | true | none/none | 155 | 29 | pid:69066 | 1172 | 1 | 13.4 |
78+
| 3 | 1055.3 | true | none/none | 155 | 29 | pid:69393 | 1162 | 1 | 13.34 |
79+
| 4 | 1049.9 | true | none/none | 155 | 29 | pid:69892 | 1169 | 1 | 13.21 |
80+
| 5 | 1043.5 | true | none/none | 155 | 29 | pid:70202 | 1101 | 1 | 13.35 |
7381

7482
## Live candidate recovery
7583

76-
- 0/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response.
84+
- 4/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response.
7785

7886
| Operation | Observed failure | Recovery response | Recovered tree |
7987
|---|---|---|---|
80-
| process-crash | process-crash/persistent-process-exited | failed | 0 nodes |
81-
| timeout | timeout/guest-read-timeout | failed | 0 nodes |
82-
| cancelled | cancelled/abort-signal | failed | 0 nodes |
83-
| stale-generation | timeout/batch-duration-limit | failed | 0 nodes |
88+
| process-crash | process-crash/guest-exited | ok | 155 nodes |
89+
| timeout | timeout/batch-duration-limit | ok | 155 nodes |
90+
| cancelled | cancelled/abort-signal | ok | 155 nodes |
91+
| stale-generation | stale-generation/target-generation-mismatch | ok | 155 nodes |
8492

85-
## Hierarchy residue
93+
## Hierarchy
8694

87-
- provider-pruned/depth observed; traversal depth is not treated as complete.
88-
- Observed traversal depth: 0; depth complete: **false**. The guest response is flat and carries typed `provider-pruned/depth` residue.
95+
- nested tree with traversal depth 29 in 5/5 samples; truncated=false.
96+
- Observed traversal depth: 29; depth complete: **true**; interpretation: nested-tree.
8997

9098
## Stretch findings
9199

@@ -95,6 +103,7 @@ Cold and cold-cold first-look measurements remain visible for diagnosis, but are
95103
- Original broad-run finding: guest-simulator-framework-bridge relaunch first look missed the 250 ms target.
96104
- Cold and cold-cold first-look measurements include Simulator, app, daemon, and runner readiness costs; they are diagnostics, not candidate-owned hard gates.
97105
- The former warm 75/150 ms and relaunch 250 ms thresholds are stretch findings under the corrected contract.
106+
- Nonresident bootstrap samples were taken on a host with 1-minute load average 13.75 on 12 cores; per-sample load is recorded with each sample.
98107

99108
## Production boundary
100109

Binary file not shown.
Binary file not shown.
Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,17 @@
11
# iOS Simulator AX bridge spike
22

3-
This bounded harness supplies the decision evidence for [#2192](https://github.com/callstack/agent-device/issues/2192). It compares the official idb `SimulatorFrameworkBridge` guest mechanism with the #2189 XCTest control baseline behind one acquisition adapter. It does not select a production backend or change daemon, runner, open, relaunch, proxy, interaction, or public CLI behavior.
3+
This bounded harness supplies the decision evidence for [#2192](https://github.com/callstack/agent-device/issues/2192). It compares an in-Simulator accessibility reader with the #2189 XCTest control baseline behind one acquisition adapter. It does not select a production backend or change daemon, runner, open, relaunch, proxy, interaction, or public CLI behavior, and nothing in it ships in the npm package.
4+
5+
## Mechanism under test
6+
7+
The guest candidate is idb v1.5.2's `Resources/SimulatorFrameworkBridge`: a 196 KB iOS-Simulator executable that reads the XCTest-shaped element tree (`XC_kAXXCAttribute*`) inside the Simulator and serves it over a UNIX socket. The spike drives it **directly from Node**:
8+
9+
- `xcrun simctl spawn <udid> SimulatorFrameworkBridge accessibility serve <socket> --idle-timeout 300 --exit-on-disconnect true` starts one private guest per session in the Simulator's launchd domain;
10+
- the host connects to the socket and exchanges 4-byte big-endian length-prefixed JSON frames;
11+
- each read is one `describe` with `snapshotTree=true` (one XCTest snapshot fetch, one Mach round trip) and `automationMode=true`, so the target exposes its accessibility server without preboot preference edits;
12+
- a known target generation reads by pid; otherwise the guest resolves the foreground app in-guest through RunningBoard.
13+
14+
No `idb_companion`, gRPC, or Python client is involved. The earlier prototype packaging (companion + Python reader) is retained only as the superseded targeted artifact.
415

516
Build the repository first:
617

@@ -9,7 +20,9 @@ pnpm install --frozen-lockfile
920
pnpm build
1021
```
1122

12-
Use a newly created, task-owned iOS Simulator. The guest candidate uses the arm64 [idb release](https://github.com/facebook/idb/releases/tag/v1.5.2) outside this repository:
23+
Obtain the guest executable from the official arm64 [idb v1.5.2 release](https://github.com/facebook/idb/releases/tag/v1.5.2) (`idb-companion.macos-arm64.tar.gz`, SHA-256 `f17b718a513931705542a7fbfa9cfc11895ee191562c9ffd2343cf7f8254bc08`; the extracted `Resources/SimulatorFrameworkBridge` has SHA-256 `3545621d2dc98de32879ebac55e8b0c33dc8eb7cc2bfbc2d0d2d21a002c8de58`). Use a task-owned iOS Simulator with the test app installed.
24+
25+
## Broad corpus
1326

1427
```sh
1528
pnpm bench:ios-ax-bridge -- \
@@ -18,33 +31,26 @@ pnpm bench:ios-ax-bridge -- \
1831
--state cold-cold,cold,warm,relaunch \
1932
--screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \
2033
--samples 20 \
21-
--apply-preferences \
22-
--guest-companion /path/to/idb_companion \
23-
--guest-python python3 \
24-
--guest-site-packages /path/to/idb-cli/libexec/lib/python3.14/site-packages \
34+
--guest-bridge /path/to/Resources/SimulatorFrameworkBridge \
2535
--out .tmp/ios-ax-bridge-spike.v1.json.gz
2636
```
2737

28-
The default candidate set, state set, screen set, and sample minimums come from the #2189 benchmark definitions. Each request carries an optional expected target generation and fixed request, response, node-count, traversal-depth, CPU, memory, and duration bounds. The guest adapter uses a newline-delimited reader, keeping one idb gRPC client and one `axbridge-persistent` reader alive across the run. Guest reads request idb's flat raw element form, preserving provider facts without importing visibility, hittability, scope, depth, or semantic compaction. Every sample keeps resource metrics and target status; each cell keeps one raw-tree exemplar with viewport, lineage, truncation, residue, and bounded diagnostics, plus separate prototype presentation measurements.
38+
The default candidate set, state set, screen set, and sample minimums come from the #2189 benchmark definitions. Each request carries an optional expected target generation and fixed request, response, node-count, traversal-depth, CPU, memory, and duration bounds. Guest reads keep the nested view hierarchy as parent-linked raw nodes with XCTest type names, labels, values, identifiers, and frames; they do not import visibility, hittability, scope, depth, or semantic compaction. Every sample keeps resource metrics and target status; each cell keeps one raw-tree exemplar with viewport, lineage, truncation, residue, and bounded diagnostics, plus separate prototype presentation measurements.
39+
40+
Hard tiers follow the maintainer-corrected #2192 contract: warm p50 < 300 ms and p95 < 500 ms per screen, relaunch p95 < 500 ms after observed app readiness. The former 75/150 ms and 250 ms values are reported as stretch findings and never decide GO/NO-GO.
2941

30-
`--apply-preferences` is the only way the experiment edits Simulator preference plists. The Simulator must be shutdown; the harness records exact plist hashes and targeted key changes, then restores the original bytes before reporting. The keys are not production defaults.
42+
`--apply-preferences` optionally runs the task-owned preboot AX preference experiment. The Simulator must be shutdown; the harness records exact plist hashes and targeted key changes, then restores the original bytes before reporting. The keys are not production defaults and the guest path does not need them.
3143

32-
The harness fails closed. It reports `NO-GO` when the guest candidate is unsupported, unavailable, unreadable, stale, over a bound, below the sample minimum, or when crash/timeout/cancellation recovery is not typed and recovered. XCTest is a control result and cannot turn a passing guest corpus into a failure. It stops before reporting timings if the fixture app cannot be prepared deterministically. The adjacent gzipped JSON and readable Markdown report are the decision artifact; no production route should be implemented from a `NO-GO` run.
44+
## Targeted evidence
3345

34-
The checked-in broad corpus predates the corrected hard-latency contract. Reproduce only the
35-
missing live bootstrap and lifecycle evidence with:
46+
The broad corpus predates the corrected hard-latency contract. The live nonresident-bootstrap and lifecycle evidence is produced with:
3647

3748
```sh
3849
pnpm bench:ios-ax-bridge:targeted -- \
3950
--udid SIMULATOR_UDID \
40-
--apply-preferences \
41-
--guest-companion /path/to/idb_companion \
42-
--guest-python python3 \
43-
--guest-site-packages /path/to/idb-cli/libexec/lib/python3.14/site-packages
51+
--guest-bridge /path/to/Resources/SimulatorFrameworkBridge
4452
```
4553

46-
This preserves the broad raw artifact and writes a narrow raw artifact plus the superseding
47-
corrected report. Each nonresident bootstrap sample re-establishes a booted Simulator and ready app
48-
before its timer, so helper teardown contention and Simulator/app readiness are outside the measured
49-
candidate boundary. Missing provider generation is emitted as typed residue; the reader never echoes
50-
an expected generation it did not observe.
54+
For each of five bootstrap samples the fixture app is relaunched, a throwaway probe bridge polls until the new app generation answers with a tree (readiness is observed, never assumed from a pid), the probe exits, and only then a fresh guest is spawned and timed to its first usable tree. Host load is recorded per sample. Recovery probes exercise process crash, timeout, cancellation, and a dead target generation through the same adapter and require a typed failure plus a usable recovered read. The run preserves the broad raw artifact, writes the narrow raw artifact, and regenerates the corrected report.
55+
56+
The harness fails closed. It reports `NO-GO` when the guest candidate is unsupported, unavailable, unreadable, stale, over a bound, below the sample minimum, or when crash/timeout/cancellation recovery is not typed and recovered. XCTest is a control result and cannot turn a passing guest corpus into a failure.

‎scripts/ios-ax-bridge-spike/adapter.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ test('control mapping preserves the producer raw node type', () => {
2828
assert.equal(result?.nodes[0]?.role, 'AXButton');
2929
});
3030

31-
test('guest adapter fails closed when the official companion is not configured', async () => {
31+
test('guest adapter fails closed when the guest bridge executable is not configured', async () => {
3232
const adapter = createGuestSimulatorFrameworkBridgeAdapter({ repoRoot: '/repo' });
3333
const result = await adapter.acquireBatch([
3434
{

‎scripts/ios-ax-bridge-spike/adapter.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,8 @@ export type AcquisitionBatchResult = Readonly<{
3333
export type AdapterOptions = Readonly<{
3434
repoRoot: string;
3535
limits?: ResourceLimits;
36-
guestCompanion?: string;
37-
guestPython?: string;
38-
guestSitePackages?: string;
36+
/** Path to the in-Simulator `SimulatorFrameworkBridge` guest executable (idb v1.5.2 Resources). */
37+
guestBridge?: string;
3938
}>;
4039

4140
export function createXCTestControlAdapter(

0 commit comments

Comments
 (0)