Skip to content

Commit 6b0b9cb

Browse files
authored
feat(ios): usbmux runner route override + #1403 transport experiment evidence (#1510)
* feat(ios): usbmux runner route override + #1403 transport experiment evidence Adds AGENT_DEVICE_IOS_RUNNER_ROUTE=usbmux, an experimental override that routes coredevice-backend physical devices' runner commands through usbmux, and simplifies the xctest branch that awaited a no-op resolveRunnerTransport. Documents the live #1403 experiment (iPhone 17 Pro, USB + Wi-Fi legs): steady-state is a wash, but the >30s-idle tax drops from ~4.5s (tunnel re-probe + session re-establish) to ~440ms because the usbmux session stays hot; CoreDevice Wi-Fi devices never appear in usbmuxd, so the verdict is usbmux-primary with network fallback rather than tunnel-code deletion. Also records the cable-out failure gap (2x45s retry hang swallowing the usbmux DEVICE_NOT_FOUND hint), which affects today's xctest backend too. * fix(ios): document daemon scoping of the usbmux route override The env is re-read per resolve but from the daemon's environment, which is captured at daemon launch — a later CLI invocation cannot flip the route on a running daemon. Correct the source comment and experiment doc, and lock the read-point semantics with a regression test.
1 parent 209ac83 commit 6b0b9cb

3 files changed

Lines changed: 163 additions & 12 deletions

File tree

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# #1403 experiment: usbmux as the primary physical-iOS runner transport
2+
3+
Live results from 2026-07-30, thymikee-iphone (iPhone 17 Pro, iOS 26.5.2, UDID
4+
00008150-001849640CF8401C), macOS host with Xcode 26.2. Method: forced-route
5+
env override `AGENT_DEVICE_IOS_RUNNER_ROUTE=usbmux` in `runner-command-route.ts`
6+
(committed with this doc), isolated `--state-dir` daemons per leg, timings
7+
from per-request `--debug` ndjson plus wall-clock. The override is
8+
**daemon-scoped**: the daemon captures its environment at launch, so setting
9+
the variable on a later CLI invocation against a running daemon silently
10+
keeps the previous route — every route leg needs a fresh isolated
11+
state-dir/daemon launched with the desired value. Every timing sample was
12+
validated against real command output after an early instrument artifact (see
13+
Pitfalls).
14+
15+
## Numbers (same build, same cable state, verified output)
16+
17+
| Scenario | network (tunnel-IP) route | usbmux route |
18+
|---|---|---|
19+
| Steady-state snapshot, cable in | 350–450 ms | 440 ms |
20+
| Snapshot after 40 s idle | **4.5–4.6 s** (tunnel re-probe + session re-establish) | **440 ms** (unchanged) |
21+
| Runner killed, next snapshot | ~4.0 s self-heal | 4.7 s self-heal |
22+
| Steady-state snapshot, Wi-Fi only (no cable) | 425–495 ms | fails (see below) |
23+
| Full lifecycle open/snapshot/tap/screenshot/close | works | works |
24+
25+
Older-build measurements (first pass) agreed in shape: first open pays one-time
26+
xctestrun prep (~13 s cold) and runner startup; fresh device resolution costs
27+
~6.7 s of devicectl listing either way.
28+
29+
## Findings
30+
31+
1. **The idle tax is the payoff, and it is bigger than expected.** Past the
32+
30 s tunnel-IP cache TTL, the network route re-probes the tunnel and
33+
re-establishes runner readiness: ~4.5 s per idle gap, every time. Over
34+
usbmux the session stays hot indefinitely (40 s gaps measured at steady
35+
440 ms). Agent workflows are exactly the idle-gappy workload that hits this
36+
tax on almost every step.
37+
2. **Steady state is a wash.** The per-command usbmuxd handshake (fresh unix
38+
socket, ListDevices, Connect) roughly cancels the network route's pooled-
39+
connection advantage; ~±90 ms either way.
40+
3. **Wi-Fi deletion is off the table.** A CoreDevice-paired, Wi-Fi-reachable
41+
iPhone (devicectl `available (paired)`, live tunnel) does NOT appear in
42+
usbmuxd at all — modern CoreDevice Wi-Fi runs over `remoted`, invisible to
43+
usbmuxd. usbmuxd listed the device only while cabled, `ConnectionType: USB`
44+
only, and the entry disappears on unplug. The CoreDevice network route must
45+
stay as the Wi-Fi fallback → this is a "usbmux-primary + network-fallback"
46+
reshuffle, not a full tunnel-code deletion.
47+
4. **Failure semantics gap (also affects today's xctest backend).** With the
48+
cable out, forced-usbmux commands hang for 2×45 s of retries and surface a
49+
generic "Runner did not accept connection" — the usbmux client's precise
50+
`DEVICE_NOT_FOUND` ("Connect the device by cable, trust this Mac…") is
51+
swallowed by `shouldRetryRunnerConnectError`. An implementation should make
52+
usbmux DEVICE_NOT_FOUND non-retryable (or fall back to the network route
53+
immediately) and preserve the hint.
54+
5. **Lock behavior is transport-independent.** A locked phone wedges AX
55+
capture/runner relaunch identically on both routes (SBMainWorkspace
56+
"Locked" denial; xcodebuild "Unlock thymikee-iphone to Continue").
57+
6. **usbmux UDID matching is exact.** usbmuxd `SerialNumber` equals the
58+
dashed hardware UDID that `DeviceInfo.id` already uses for devicectl-listed
59+
devices; DeviceID (10) is the mux handle. Multi-device disambiguation is
60+
structural. (Multi-attached-device matrix not exercised: one device only.)
61+
62+
## Implied design (matches the issue's desired outcome, minus full deletion)
63+
64+
- One private route resolver: physical devices try usbmux first; on usbmux
65+
DEVICE_NOT_FOUND (not attached via USB) resolve the CoreDevice tunnel and
66+
use the network route. Tunnel cache/invalidations stay but only run on the
67+
fallback path, so cabled devices never pay the probe/TTL tax.
68+
- Failure semantics: usbmux DEVICE_NOT_FOUND → immediate fallback attempt;
69+
if the fallback also fails, surface the usbmux hint (cable/trust/unlock).
70+
- The xctest backend keeps usbmux-only (CoreDevice unavailable by definition)
71+
but needs the same non-retryable classification for DEVICE_NOT_FOUND.
72+
73+
## Pitfalls for whoever implements/re-measures
74+
75+
- `bin/agent-device.mjs` runs `dist/`, not `src/`. A stale dist silently ran
76+
the first "usbmux leg" on stock network code while the env override sat
77+
inert in the daemon (numbers looked plausible; only the cable-pull test
78+
exposed it). Rebuild and verify the daemon process (`ps eww <pid>`, dist
79+
mtime, and a behavioral probe) before trusting route attribution.
80+
- `/usr/bin/time` on a failing command still prints a small `real` — a fast
81+
error latency is indistinguishable from a fast success unless output is
82+
asserted. Runner-lease contention ("already owned by another daemon")
83+
produced exactly this artifact.
84+
- The route override is daemon-scoped (env captured at daemon launch). Never
85+
flip `AGENT_DEVICE_IOS_RUNNER_ROUTE` between commands against the same
86+
daemon and expect the route to change — start a fresh isolated
87+
state-dir/daemon per leg, then verify with `ps eww <daemon-pid>`.
88+
- Sessions are cwd-scoped; run every command of a leg from the same cwd.
89+
- Auto-Lock must be Never during measurement or the runner dies mid-series.

‎src/platforms/apple/core/__tests__/runner-transport.test.ts‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ beforeEach(() => {
6060

6161
afterEach(() => {
6262
vi.unstubAllGlobals();
63+
vi.unstubAllEnvs();
6364
});
6465

6566
test('waitForRunner propagates request cancellation without fallback', async () => {
@@ -224,6 +225,55 @@ test('sendRunnerCommandOnce routes xctest physical devices through usbmux', asyn
224225
]);
225226
});
226227

228+
test('sendRunnerCommandOnce routes coredevice physical devices through usbmux when overridden', async () => {
229+
vi.stubEnv('AGENT_DEVICE_IOS_RUNNER_ROUTE', 'usbmux');
230+
const fetchMock = vi.fn();
231+
vi.stubGlobal('fetch', fetchMock);
232+
233+
const response = await sendRunnerCommandOnce(iosDevice, 8100, { command: 'uptime' }, 5_000);
234+
235+
assert.equal(response.status, 200);
236+
assert.equal(fetchMock.mock.calls.length, 0);
237+
assert.equal(mockRunCmd.mock.calls.length, 0);
238+
assert.deepEqual(mockUsbmuxPostCommand.mock.calls[0]?.slice(0, 3), [
239+
iosDevice.id,
240+
8100,
241+
{ command: 'uptime' },
242+
]);
243+
});
244+
245+
test('runner route override follows the daemon process env across resolves', async () => {
246+
// The override is daemon-scoped: resolves re-read the daemon's process env,
247+
// which is fixed at daemon launch. Clearing it here models a fresh daemon
248+
// launched without the variable — the only way the route can change.
249+
vi.stubEnv('AGENT_DEVICE_IOS_RUNNER_ROUTE', 'usbmux');
250+
stubSuccessfulFetch();
251+
mockRunCmd.mockImplementation(async () => ({ exitCode: 1, stdout: '', stderr: '' }));
252+
253+
await sendRunnerCommandOnce(iosDevice, 8100, { command: 'uptime' }, 5_000);
254+
assert.equal(mockUsbmuxPostCommand.mock.calls.length, 1);
255+
assert.equal(vi.mocked(fetch).mock.calls.length, 0);
256+
257+
vi.unstubAllEnvs();
258+
await sendRunnerCommandOnce(iosDevice, 8100, { command: 'uptime' }, 5_000);
259+
260+
assert.equal(mockUsbmuxPostCommand.mock.calls.length, 1);
261+
assert.equal(vi.mocked(fetch).mock.calls[0]?.[0], 'http://127.0.0.1:8100/command');
262+
});
263+
264+
test('waitForRunner routes coredevice physical devices through usbmux when overridden', async () => {
265+
vi.stubEnv('AGENT_DEVICE_IOS_RUNNER_ROUTE', 'usbmux');
266+
const fetchMock = vi.fn();
267+
vi.stubGlobal('fetch', fetchMock);
268+
269+
const response = await waitForRunner(iosDevice, 8100, { command: 'snapshot' }, undefined, 5_000);
270+
271+
assert.equal(response.status, 200);
272+
assert.equal(fetchMock.mock.calls.length, 0);
273+
assert.equal(mockRunCmd.mock.calls.length, 0);
274+
assert.equal(mockUsbmuxPostCommand.mock.calls.length, 1);
275+
});
276+
227277
function stubSuccessfulFetch(): void {
228278
vi.stubGlobal(
229279
'fetch',

‎src/platforms/apple/core/runner/runner-command-route.ts‎

Lines changed: 24 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,15 @@ import { resolveIosPhysicalDeviceControl } from '../physical-device-control.ts';
33

44
const RUNNER_DEVICE_TUNNEL_IP_CACHE_TTL_MS = 30_000;
55

6+
/**
7+
* Experimental override for #1403: forces physical-device runner commands
8+
* through usbmux regardless of backend. Daemon-scoped: the value is re-read
9+
* on every resolve, but from the daemon's environment, which is captured at
10+
* daemon launch — a later CLI invocation cannot change it. Use a fresh
11+
* isolated state-dir/daemon per experiment route leg.
12+
*/
13+
const RUNNER_ROUTE_OVERRIDE_ENV = 'AGENT_DEVICE_IOS_RUNNER_ROUTE';
14+
615
type DeviceTunnelIpCacheEntry = {
716
ip: string;
817
expiresAt: number;
@@ -33,13 +42,8 @@ export function createRunnerCommandRouteResolver(device: DeviceInfo, port: numbe
3342
return buildNetworkRoute(device, port, null, false);
3443
}
3544
const control = resolveIosPhysicalDeviceControl(device);
36-
if (control.backend === 'xctest') {
37-
const transport = await control.resolveRunnerTransport(device, timeoutBudgetMs);
38-
return {
39-
kind: transport.kind,
40-
endpoints: [`usbmux://${device.id}:${port}/command`],
41-
cachedTunnelIp: false,
42-
};
45+
if (control.backend === 'xctest' || readRunnerRouteOverride() === 'usbmux') {
46+
return buildUsbmuxRoute(device, port);
4347
}
4448
if (!forceRefresh) {
4549
const cached = readDeviceTunnelIpCache(device.id);
@@ -50,11 +54,7 @@ export function createRunnerCommandRouteResolver(device: DeviceInfo, port: numbe
5054
}
5155
const transport = await control.resolveRunnerTransport(device, timeoutBudgetMs);
5256
if (transport.kind === 'usbmux') {
53-
return {
54-
kind: 'usbmux',
55-
endpoints: [`usbmux://${device.id}:${port}/command`],
56-
cachedTunnelIp: false,
57-
};
57+
return buildUsbmuxRoute(device, port);
5858
}
5959
const tunnelIp = transport.tunnelIp;
6060
requestTunnelIp = tunnelIp;
@@ -75,6 +75,18 @@ export function clearDeviceTunnelIpCache(): void {
7575
deviceTunnelIpCache.clear();
7676
}
7777

78+
function readRunnerRouteOverride(): 'usbmux' | null {
79+
return process.env[RUNNER_ROUTE_OVERRIDE_ENV]?.trim() === 'usbmux' ? 'usbmux' : null;
80+
}
81+
82+
function buildUsbmuxRoute(device: DeviceInfo, port: number): RunnerCommandRoute {
83+
return {
84+
kind: 'usbmux',
85+
endpoints: [`usbmux://${device.id}:${port}/command`],
86+
cachedTunnelIp: false,
87+
};
88+
}
89+
7890
function buildNetworkRoute(
7991
device: DeviceInfo,
8092
port: number,

0 commit comments

Comments
 (0)