Skip to content

feat(ios): physical iOS 27 support - #538

Closed
latekvo wants to merge 86 commits into
mainfrom
feat/physical-ios-via-simserver
Closed

latekvo wants to merge 86 commits into
mainfrom
feat/physical-ios-via-simserver

Conversation

@latekvo

@latekvo latekvo commented Jul 21, 2026 •

Copy link
Copy Markdown
Member

What

Adds physical-iPhone control to argent by routing it through radon's native ios_device simulator-server controller (software-mansion/radon#146). A tethered iPhone becomes just another sim-server subcommand, exactly like an iOS simulator or a physical Android phone.

This supersedes #369 (now closed), which built a CoreDevice engine inside argent itself: a base64-embedded pymobiledevice3 Python sidecar plus a privileged root tunnel, and one-shot screenshots with no live streaming or recording. Putting the engine in the component that already speaks device wire protocols lets physical iOS reuse radon's media pipeline and keeps argent to routing and discovery. None of #369's Python or root-tunnel apparatus exists on main, so nothing of it is carried here - this PR is the complete feature against main on its own.

What argent does

  • Discovery. list-devices gains connected physical iPhones, found via xcrun devicectl list devices (listIosDevices / parsePhysicalIosDevices). Only currently-reachable iOS devices are kept - devicectl also enumerates the host's simulators as platform: "iOS", so entries are filtered on the physical ECID UDID shape and on carrying a transport. iOS entries now carry kind ("simulator" / "device") and physical ones a productType. Discovery is skipped entirely unless the flag is on.
  • Classification. classifyDevice recognises the physical UDID shape (8hex-16hex) as iOS, and resolveDevice resolves it to kind: "device" - the same shape-based split Android already uses for emulator vs phone. isPhysicalIosUdid / isPhysicalIos are the shared predicates.
  • subcommandForDevice returns ios_device for a physical iPhone, mirroring android_device. The normal sim-server spawn / api_ready / stdin-command path drives it from there.
  • Input and capture over the sim-server. gesture-tap, gesture-swipe, keyboard, button, rotate, screenshot and screen-recording-start / -stop resolve a physical iPhone to simulatorServerRef and go through the shared sendCommand / httpScreenshot transport, with no special-cased branch. gesture-swipe settles on device, since the controller replays the eased Move samples. button rejects appSwitch and actionButton, which have no hardware HID equivalent, before spawning anything. screenshot skips its tvOS probe for a hardware UDID, which can never be a sim runtime.
  • App lifecycle over devicectl. launch-app, restart-app, reinstall-app and open-url shell xcrun devicectl for kind: "device" instead of simctl. These don't go through the sim-server factory, so each calls the flag assertion directly.
  • describe reads the on-screen accessibility tree from the sim-server's /api/ax-tree endpoint - radon's native axAudit over CoreDevice DTX - through ios-coredevice-ax-adapter.ts. One USB session drives both input and describe, so there's no second tunnel and no pmd3 involved. The fetch is bounded by an AbortSignal.timeout, like the screenshot path.
    • Frames are approximate, and results are not comparable between calls. radon#146 sends captions + reading order only; Apple exposes no per-element geometry app-free on hardware. The adapter therefore interpolates every frame (full-width rows, top to bottom). Separately, the device's own VoiceOver cursor rotates the element order by one on each read, so a given element's synthesised frame changes between calls and two describe results cannot be diffed. The describe hint, README and tool docs all say so. The adapter still honours real rects if a future sim-server sends any, so landing geometry is a producer-side change.
  • Opt-in stays behind the physical-ios-devices flag. The sim-server factory is the chokepoint every CoreDevice-backed tool resolves through, raising CORE_DEVICE_FLAG_DISABLED when it's off; list-devices checks the flag before discovering at all, and the four devicectl tools call the same assertion directly.
  • boot-device on a physical UDID has nothing to boot, so it resolves (spawns) the sim-server as an explicit "prepare" step - the flag gate and an unauthorized device surface there rather than on the first tap.
  • What stays off. gesture-pinch, gesture-rotate, tv-remote, the seven native-* devtools and the native-profiler-* / profiler query tools set apple.device: false, so the capability gate rejects a physical UDID with a 400 rather than failing deep. Each declaration records why: the native-devtools family injects its dylib with DYLD via simctl spawn, which cannot reach a signed app on hardware; the two multi-touch gestures need two simultaneous contacts and the CoreDevice digitizer is single-contact; native-profiler-* resolves its target through simctl spawn / simctl listapps. (paste is not in this list: it is absent from setup-registry.ts on every platform, so its gate never runs at all.) gesture-custom stays enabled and rejects only its own two-contact half per request. ax-service and native-devtools also refuse kind: "device" at the blueprint, as a backstop for direct resolution.
  • README gains a "Physical iOS devices (experimental)" section covering requirements, setup and the current limitations.

Deliberately left out

  • Element geometry for describe. Tracked on radon#146; until then every frame is interpolated.
  • Multi-touch. Pinch and rotate need a second contact the CoreDevice digitizer does not expose.
  • Distribution. Nothing - this needs no argent-side step. radon#146 merged on 2026-07-28 and ships ios_device in the macOS argent sim-server binary; scripts/download-simulator-server.sh pulls the rolling radon-main tag rather than a pinned version, so there is no binary version to bump. The published simulator-server-argent-macos asset (rebuilt 2026-07-29 15:21Z) carries ios_device, ios_axtree, /api/ax-tree and device_controller/ios_device/axaudit.rs, so a release cut from here bundles the controller. An older bundled binary still exits 1 with "Unrecognized argument: ios_device"; that is classified and told to upgrade rather than reading as a crash.

Verification

Hardware (physical iPhone, iOS 27, USB), end to end through argent's real tool-server (POST /tools/*), pointed at a local radon build via ARGENT_SIMULATOR_SERVER_DIR:

  • screenshot on home and Settings, pixel-verified as live frames (a Settings frame differs from home).
  • gesture-tap and gesture-swipe change the on-screen tree; button home returns to home; button appSwitch|actionButton|back reject cleanly with a 400 and the driver survives and still screenshots.
  • describe over /api/ax-tree reflects the frontmost app (home vs Settings), with source: coredevice-ax. Repeated calls return the same elements rotated by one - the VoiceOver-cursor behaviour documented above, not a stability guarantee.
  • Adversarial inputs (out-of-range normalized taps, a gated tool) are handled without killing the driver, and no pmd3 processes are spawned at any point.
  • An adversarial E2E sweep converged clean over consecutive runs.

Local checks on the current head: tsc --build, tsc -p packages/tool-server/tsconfig.test.json, eslint . --max-warnings 0, prettier --check, npm run typecheck:scripts, npm run test:scripts (59 pass) and the full vitest suite (4800 pass, 1 skipped).

Review pass

A review of this branch found three defects, each reproduced before being fixed and pinned by a test that fails on the un-changed code:

  • A malformed /api/ax-tree payload crashed the describe adapter with a bare TypeError - a 500 carrying no error_code, which reads as an argent bug rather than a device answering out of contract. {"elements":{}} and {"elements":"str"} gave "els.map is not a function", [null] gave "Cannot read properties of null", a numeric caption gave "caption.split is not a function", and a non-numeric screen width escaped the sw <= 0 guard as NaN and surfaced a raw ZodError. The ?? [] only ever guarded null/undefined. The response is now validated where it enters, the way describe's other adapter validates its own raw payload.
  • open-url reported success for a URL it never opened. The url was passed straight through as the last argv element of xcrun devicectl device process openURL; that command with --help in the url position exits 0 and prints help, so the tool returned opened: true. The schema now requires an RFC 3986 scheme - the same guard launch-app already applies to bundleId, and one that also rejects the schemeless input no backend could have opened.
  • A bundled simulator-server predating the controller looked like a crash. The released binary exits 1 with "Unrecognized argument: ios_device"; that surfaced as a generic "exited before becoming ready", bucketed with genuine crashes, and never said the binary ships inside argent. It now names the cause, says to upgrade, and has its own failure code.

The same pass compared each new physical-iOS path against its already-pinned physical-Android twin. The capability gate was the largest gap: the simulator-only list was hand-written under a comment claiming it was "enumerated rather than sampled", but settings-permissions was missing, and giving it apple.device: true left the whole suite green while a physical iPhone walked into simctl privacy with a hardware UDID. That set is now derived from the registry, so the claim holds by construction and future tools are covered without being listed. Also newly pinned, each against the mutation it catches: the shutdownDevice / shutdownOwnedDevice physical-iPhone refusal, getSimulatorRuntimeKind's hardware short-circuit, the ax-service blueprint backstop, and httpAxTree's error branch (deleting it turned an HTTP 500 into an empty element list, so describe reported "the screen may be off or locked" for a read that never happened).

Second review pass

A second pass reviewed the branch again, including the first pass's own fixes. Same bar: every defect reproduced before being touched, the same reproduction re-run after, and a test that fails on the un-fixed code.

Reported success for something that did not happen, or could not:

  • Flows accepted a physical iPhone and then failed every step. Treating state: "connected" as ready made an attached iPhone a run target - explicitly, and as auto-detection's single match - but fetchFlowTree sends every iOS device to the native view hierarchy, which nativeDevtoolsBlueprint refuses for kind: "device". Every selector step failed with a devtools error, and a launch step answered "restart the argent server and try again", advice that can never work. The runner now refuses a hardware udid at device resolution with the permanent reason, and auto-detection ignores it rather than turning a lone simulator into "2 booted devices matched".
  • await-screen-idle could never settle on a screen of 120+ elements. Sorting the signature cancels the read's rotation only while the whole screen fits in one read; describe asks for at most 120 and the walk advances the VoiceOver cursor per read, so past that each poll covers a different window. A still screen of 121 elements took 76 polls, spent the full 15s, and returned settled: false saying nothing. It now recognises a truncated read and answers immediately with the cause.
  • httpAxTree and httpScreenshot died with a bare TypeError on a JSON null body - including on the 5xx responses a server that cannot answer is likeliest to send. That is the exact class the axAudit schema was added to prevent; the schema ran after the in-band { error } read.
  • Every devicectl failure was reported as a signing problem and the child's own output was discarded, so a locked phone was sent to re-sign a bundle that was fine. subprocessFailureMetadata records only exit code and signal, so nothing else carried devicectl's explanation.
  • shutdownOwnedDevice reached for a physical Android phone while the comment beside it and the test covering it both described a physical-Android exclusion that did not exist.

Reachable but wrong:

  • describe discarded the VoiceOver state tokens it strips from the caption, so an enabled and a disabled control came back byte-identical - while the hint promises the traits are exact. They now land on disabled / selected.
  • gesture-custom resolved (spawned) the CoreDevice session before rejecting a two-contact request, so a device whose session cannot start answered with a transport error instead of "this gesture needs two contacts" - and falsified run-sequence's own new comment.
  • await-ui-element was reachable on hardware unadapted: the 5s simulator default against ~2s reads, and a description promising frame and reading-order semantics that do not hold there.
  • profiler-load's load_native mode built a NativeProfilerSession for a hardware udid that every reader of it rejects.
  • The flow refusal answered HTTP 500 where every sibling refusal answers 400.
  • The caption parser promoted a trait to an element's value on an all-trait caption ("Dimmed, Button" → { label: "Dimmed", value: "Button" }).

Docs that contradicted the code, each now derived or pinned: the argent-screen-recording skill listing physical iPhones as unsupported (the skill is what an agent reads before recording, so the capability underneath was never consulted); three separate stale tool enumerations; a claimed gating rule the gate does not implement; four mutually inconsistent accounts of which tools shell devicectl; and a "saves a simctl probe" rationale for a guard that saves nothing.

Coverage. A mutation sweep found the physical-iOS opt-in gate itself unpinned - deleting assertPhysicalIosEnabled from simulatorServerRef left all 3176 tests green while taking the flag off screenshot, gesture-*, keyboard, button, rotate, describe, screen-recording and boot-device at once. Also newly pinned, each against the mutation that exposed it: list-devices' flag gate, kind tag and readiness rank; listIosDevices' darwin guard, --quiet and killSignal (its simctl twin pins all three, but that file's filter keys on simctl and could never see a devicectl spawn); boot-device's force recycling and its await; describe's empty and truncated hints, element ceiling and read bound; the placeholder-frames warning; the second-touch guard's y2 half; and the adapter's clamp, arity guard and value-split restriction.

Local: tsc --build, the test tsconfig, eslint . --max-warnings 0, prettier --check, and the full vitest suite (3210 pass, 1 skipped). All CI checks green, including macOS + iOS E2E.

radon#146 has landed (merged 2026-07-28), so this no longer waits on it. Its idevice dependency is pinned pre-1.0 (=0.1.65) - flagged there, and now a shipped dependency rather than an open question.

latekvo and others added 30 commits June 19, 2026 13:44
Drive a real iPhone (iOS 27+) app-free via Apple's CoreDevice remote-control
services through pymobiledevice3 — no on-device app, no WebDriverAgent. Gated
behind the `physical-ios-devices` flag.

- Discovery: classify physical UDIDs (8hex-16hex) as ios/device; list connected
  iPhones via `xcrun devicectl`, flag-gated.
- CoreDevice backend (blueprints/core-device.ts): resolves the tunneld RSD
  endpoint, ensures the DDI is mounted, shells to pymobiledevice3 for screenshot,
  tap (dwell-drag — a zero-dwell tap is ignored by iOS), swipe, and buttons.
- Branded privilege escalation: the tunnel needs root once to create its network
  interface. Instead of manual sudo, Argent auto-starts it via a signed macOS
  host helper (argent-device-auth) that shows the standard auth modal branded as
  "Argent" with the Argent icon (Authorization Services + dlsym of
  AuthorizationExecuteWithPrivileges). Falls back to the osascript admin prompt,
  then manual-sudo instructions (headless). HOME is pinned so tunneld finds its
  pairing records under the privileged-exec environment.
- Wire screenshot/gesture-tap/gesture-swipe/button to the backend; launch-app via
  devicectl; boot-device starts the tunnel (branded prompt). Unsupported tools on
  physical iOS are cleanly rejected (simulator-server / ax-service /
  native-devtools guards; open-url/reinstall/restart handlers).
- Helper resolvers + Argent icon in @argent/native-devtools-ios;
  download-native-binaries.sh fetches the signed helper (optional until
  published); bundle-tools copies it for distribution. Helper source + signing
  live in argent-private.
- Tests for classification, devicectl parsing, coord mapping, and privileged
  command building. Overrides: ARGENT_PYMOBILEDEVICE3, ARGENT_PMD3_TUNNELD_PORT,
  ARGENT_DEVICE_AUTH_HELPER, ARGENT_DEVICE_ICON.
…y target list

The preview / Argent Lens window streams frames over simulator-server, which
refuses physical iOS devices (kind === "device", driven over CoreDevice). With
the physical-ios-devices flag on, a connected iPhone leaked into
GET /preview/simulators because the iOS branch matched any platform === "ios"
without checking kind — surfacing a target the UI can't drive (and emitting an
entry with no runtime under a type that asserted runtime: string).

- Exclude physical iOS (kind === "device") from the preview list, mirroring the
  existing Chromium exclusion.
- Consume list-devices' own exported ListDevicesResult type instead of a
  hand-rolled copy that had already drifted from it (the soundness hole tsc
  couldn't catch because invokeTool<T> takes a caller-supplied generic).
- Extract the mapping into a pure, exported devicesToPreviewEntries() and add
  regression tests asserting physical iOS + Chromium are excluded while the
  simulator and Android emulator are kept.
…app behavior

list-devices description and the simulator-server guard message listed
screenshot/gesture-tap/gesture-swipe/button but omitted launch-app, which
is implemented for physical iOS via devicectl. The README limitations list
omitted restart-app, which (like open-url/reinstall-app) returns a clear
"not supported" error on physical iOS.
…ice-support

# Conflicts:
#	packages/tool-server/src/preview.ts
#	packages/tool-server/src/tools/open-url/platforms/ios.ts
…lity matrix

Review follow-ups on the physical-iOS CoreDevice feature:

- discovery: parsePhysicalIosDevices surfaced every host iOS simulator as a
  phantom physical device (devicectl lists simulators with transportType
  "sameMachine"); gate on the physical ECID UDID shape, which also aligns
  discovery with classifyDevice's routing. Verified against real devicectl JSON
  and a connected iPhone (returns only the device, not the host's simulators).
- run-sequence: was eagerly resolving simulator-server for a physical iPhone,
  failing the whole call before step 1; hold no eager service so per-step
  CoreDevice routing works.
- describe: swallowed the simulator-only guards and returned an empty tree with
  a misleading "reboot the simulator" hint; reject with a clear error instead.
- swipe: command timeout was a fixed 15s, killing long swipes mid-drag; scale
  the timeout with the requested duration and clamp degenerate durations
  (extracted swipeDragParams, unit-tested).
- native-profiler: guard the session blueprint so a physical iPhone isn't driven
  via simulator-only simctl (which mislabeled it an unresponsive "simulator").
- capability matrix: tools whose iOS backend is simulator-only (keyboard, paste,
  rotate, gesture-pinch/rotate/custom, screenshot-diff, native-devtools-*,
  native-profiler-start) now declare apple.device:false, so physical iOS is
  rejected with a clean 400 at the capability gate rather than a 500 from a deep
  service guard. open-url/reinstall-app/restart-app throw UnsupportedOperationError.
- launch-app: wrap the devicectl launch so "app not installed/signed" surfaces a
  clean message instead of a raw subprocess blob.
- core-device: check the feature flag before the pymobiledevice3 setup probe;
  echo a custom tunneld port in the manual-fallback hint.
- docs: flags description and README now match the supported tool set and the
  auto-start prompt reality.

Verified end-to-end on a physical iPhone (iOS 27): discovery, tunnel
auto-escalation, screenshot (1179x2556), and swipe.
…e entry

The previous commit guarded the shared native-profiler-session blueprint
against physical iOS, but that blueprint also backs the device-agnostic
analysis tools (profiler-stack-query / profiler-combined-report / -analyze),
which profiler-query-android-capability.test.ts intentionally keeps supported
on Apple device. The guard made those analysis tools 500 on a physical-iOS
device_id.

Scope the fix to where the actual bug is: only native-profiler-start enumerates
running processes via simulator-only `simctl` (mislabeling a real iPhone as an
unresponsive "simulator"), and it already declares apple.device:false, so the
capability gate rejects it cleanly (400) before execute() runs. Drop the
redundant session-blueprint guard; the analysis tools keep their cross-device
capability. Also formats physical-ios-followups.test.ts.
launch-app drives a physical iPhone via `xcrun devicectl` directly rather than
the CoreDevice service, so its handler never hit the flag check that gates every
other physical-iOS operation — with `physical-ios-devices` off, launch-app on a
physical UDID still launched the app. Extract the gate into a shared
assertPhysicalIosEnabled() (reused by the CoreDevice factory and tunnel start)
and call it from the launch-app physical-iOS branch before shelling devicectl.

Add coverage: the launch-app flag gate, assertPhysicalIosEnabled, and the
gesture-swipe physical-iOS routing (coords + duration forwarding).
… as unsupported

The README limitations section listed only the interaction/lifecycle tools that
reject on physical iOS; align it with the actual behavior by also noting that
describe (accessibility), the native-* inspection tools, native-profiler-*, and
screenshot-diff are simulator-only.
…; cover pmd3 argv

On iOS 18-26 Apple gates host-driven input (the CoreDevice "remote control"
services behind tap/swipe/button) to iOS 27+, so the device is discovered and
advertised as tap-capable, then fails mid-session with a raw `CoreDeviceError
9021`. Detect that specific failure in `conciseError` (matched against
pymobiledevice3's stderr/stdout only, never the argv-echoing execFile message,
so a HID coordinate that contains "9021" can't false-positive) and surface an
actionable message that points at iOS 27 and notes screenshots still work.
`screen-capture` is not gated, so the screenshot path is untouched.

Also note the physical-iOS/CoreDevice routing in the gesture-tap and
gesture-swipe descriptions, and add a focused test that drives the CoreDevice
factory through a mocked execFile boundary to pin the exact pmd3 argv for tap,
button, screenshot, and swipe (previously untested — every other physical-iOS
test mocks the CoreDeviceApi wholesale) plus the 9021 translation.
…ice-support

# Conflicts:
#	packages/tool-server/src/preview.ts
…quivalent

button's services() resolved coreDeviceRef() unconditionally for any
physical-iOS button press, before execute() validates whether the
button is even supported. Since the registry resolves services()
before execute(), pressing appSwitch/actionButton on a physical
iPhone triggered CoreDevice's tunnel-setup side effects (possibly a
macOS admin prompt) just to reject the button afterward. Skip
resolving the service when the button has no CoreDevice HID mapping.
…essage accuracy

core-device.ts threw plain Error at all 12 sites, and ax-service.ts /
simulator-server.ts's own new physical-iOS rejection branches did too
- unlike every sibling blueprint, which uses FailureError with a
dedicated error_code. Every physical-iOS failure (auth declined, no
tunnel, missing pymobiledevice3, iOS<27 gate, flag disabled) was
landing in telemetry as generic REGISTRY_TOOL_FAILURE_UNCLASSIFIED,
indistinguishable from any other crash. Add CORE_DEVICE_* /
AX_PHYSICAL_DEVICE_UNSUPPORTED / SIMULATOR_SERVER_PHYSICAL_DEVICE_UNSUPPORTED
failure codes and convert every throw site.

Also, three smaller correctness fixes found in the same pass:
- resolveTunnel's `tunnel-port == null` check let a port of 0 through
  as "present", inconsistent with the sibling tunnel-address check
  which treats any falsy value as missing.
- The iOS<27 gate message claimed "tap/swipe/button" all require
  iOS 27+, but this is hardware-verified false: screenshot and
  hardware buttons work below iOS 27, only touch (tap/swipe) is
  actually gated.
- conciseError's line-matching heuristic could pick a misleading
  rich-boxed Python source-context line (e.g. "if rsd is not None:")
  instead of the real exception (e.g. "OSError: No route to host")
  from a stale-tunnel traceback, and searched execFile's synthesized
  "Command failed: <argv>" message with equal priority to the real
  stderr/stdout output.
… too

native-profiler-start already rejects physical iOS (apple.device:false)
since live capture uses simctl process enumeration that mislabels a
real iPhone as a simulator. stop/analyze were left at device:true, so
- since a session can never exist for physical iOS - they always fell
through to a generic "no active session" error instead of the same
clean, actionable rejection. Also fixes the README's "native-profiler-*"
claim, which the stop/analyze gap made inaccurate, and two other
accuracy issues found in the same pass: the iOS<27 limitation
overstated to include hardware buttons, and a note that launch-app
(devicectl) doesn't need the CoreDevice tunnel other tools require.
# Conflicts:
#	packages/argent/scripts/bundle-tools.cjs
#	packages/native-devtools-ios/src/index.ts
#	packages/tool-server/src/blueprints/native-devtools.ts
#	packages/tool-server/src/blueprints/simulator-server.ts
#	packages/tool-server/src/tools/describe/platforms/ios/index.ts
#	packages/tool-server/src/tools/devices/boot-device.ts
#	packages/tool-server/src/tools/devices/list-devices.ts
#	packages/tool-server/src/tools/gesture-custom/index.ts
#	packages/tool-server/src/tools/gesture-pinch/index.ts
#	packages/tool-server/src/tools/gesture-rotate/index.ts
#	packages/tool-server/src/tools/keyboard/index.ts
#	packages/tool-server/src/tools/launch-app/index.ts
#	packages/tool-server/src/tools/launch-app/platforms/ios.ts
#	packages/tool-server/src/tools/native-devtools/native-describe-screen.ts
#	packages/tool-server/src/tools/native-devtools/native-devtools-status.ts
#	packages/tool-server/src/tools/native-devtools/native-find-views.ts
#	packages/tool-server/src/tools/native-devtools/native-full-hierarchy.ts
#	packages/tool-server/src/tools/native-devtools/native-network-logs.ts
#	packages/tool-server/src/tools/native-devtools/native-user-interactable-view-at-point.ts
#	packages/tool-server/src/tools/native-devtools/native-view-at-point.ts
#	packages/tool-server/src/tools/paste/index.ts
#	packages/tool-server/src/tools/restart-app/index.ts
#	packages/tool-server/src/tools/restart-app/platforms/ios.ts
#	packages/tool-server/src/tools/rotate/index.ts
#	packages/tool-server/src/tools/run-sequence/index.ts
#	packages/tool-server/src/tools/screenshot/index.ts
#	packages/tool-server/src/utils/device-info.ts
#	packages/tool-server/src/utils/ios-devices.ts
#	scripts/download-native-binaries.sh
# Conflicts:
#	packages/tool-server/src/preview.ts
The app-free CoreDevice accessibility path was investigated on a physical
iPhone (iOS 27). The on-device accessibility tree is served by CoreDevice's
axAuditDaemon, but Apple gates it: the DTX service
`com.apple.accessibility.axAuditDaemon.remoteserver.shim.remote` requires the
`com.apple.mobile.lockdown.remote.trusted` entitlement — over the developer
(untrusted) CoreDevice tunnel pymobiledevice3 forms, the daemon accepts the
socket but terminates it on the first request (every audit selector, and even
the standard DTX capability handshake). The RemoteXPC replacement
`…axAuditDaemon.remoteAXService` requires `AppleInternal`, unreachable by any
third party. DTX transport itself works over the same tunnel
(`developer dvt proclist` succeeds), so this is Apple's auth wall, not a
transport gap. Screenshot (`canViewDeviceDisplay`) remains the only app-free
screen capture.

Replace the imprecise describe rejection ("no on-device accessibility/describe
path") and the ax-service backstop's "not supported yet" with accurate,
evidence-based messages, and split describe out of the README "not yet" list.
Drive a physical iPhone through one long-lived pymobiledevice3 process per
device instead of spawning the CLI per interaction. Each `pymobiledevice3`
invocation cost ~0.8s (~0.5s just `import pymobiledevice3`); the sidecar
connects the RSD tunnel and opens the HID/screenshot services once, so a
warm tap/button/swipe is now ~150-320ms (5x) — verified live on an iPhone 15
(iOS 27) through the tool-server. The agent is a small Python program,
base64-embedded in coredevice-agent.ts (survives esbuild, no build-pipeline
wiring), materialized to a temp file and run with pmd3's own venv interpreter;
it speaks newline-delimited JSON over stdio. core-device.ts now maps its
responses (incl. the iOS-27 9021 gate) to FailureErrors and tears the process
down on dispose.

Also make `describe` work on a physical iPhone: it returns the SpringBoard
home-screen layout (app icons + dock) via CoreDevice's springboardservices —
the only app-free structured screen data on a real device. Icon frames are
derived from the home-screen grid (row-major first-fit so multi-cell widgets
don't shift the icons after them) and are approximate; the hint says to
confirm with screenshot and that in-app content isn't reachable. Verified the
describe→tap loop live (tapped Settings via describe's frame; it opened).

In-app accessibility stays Apple-gated: the axAuditDaemon DTX service drops
the connection on the first message over the developer tunnel, and its RemoteXPC
replacement needs the AppleInternal entitlement.

Replaces the per-call argv test with sidecar/adapter/agent-protocol coverage.
Replaces the SpringBoard home-screen-only describe with the device's live
on-screen accessibility tree, read app-free over CoreDevice via the iOS-26+
axAudit service. Works in ANY app and on the home screen (the same VoiceOver
walk): labels, values, traits (mapped to roles) and reading order are exact.

The unlock is the RSDCheckin handshake iOS 26 added to RSD DTX services. pmd3's
DtxServiceProvider opens RSD services with a raw connect and skips RSDCheckin,
so iOS 26/27 dropped the accessibility daemon on the first byte (which had read
as an "Apple wall"); the sidecar monkeypatches the open to go through
start_lockdown_service, which performs the checkin. Verified live on iPhone/iOS
27: describe Settings -> read the "Other…" button's frame -> gesture-tap it ->
the Join-Wi-Fi form opened. Home screen lists every app + widget.

Frames: Apple exposes no per-element geometry on hardware, so exact rects come
only from the accessibility audit (a subset of elements, correlated to the tree
by element id); the rest are interpolated from reading-order neighbours -
enough to tap a list row, with a hint to confirm precise taps via screenshot.
Pixel-exact in-app frames would need an on-device XCUITest runner (code-signing).

- sidecar: RSDCheckin fix + an `axtree` op (iter_elements walk + run_audit rects
  + screen size), replacing `homescreen`.
- new ios-coredevice-ax-adapter (caption->role/label parse, rect normalize,
  frame interpolation); removes the springboard adapter. Source: coredevice-ax.
…ice-support

Conflict in gesture-swipe: main (#429) added the `settle` momentum-free swipe
while this branch added CoreDevice routing for physical iPhones.

The two features can't simply coexist. `settle` is built from an eased train of
individual touch events sent to the simulator-server; CoreDevice instead performs
one pymobiledevice3 HID drag whose trajectory is a fixed linear interpolation
over (steps, duration), so the per-sample easing cannot be expressed through it.
Silently dropping the flag would return a flinging swipe while the caller believes
it scrolled a deterministic distance.

So: keep main's ease-out for simulators/emulators, keep CoreDevice routing for
physical iOS, and reject settle:true on a physical iPhone with an
UnsupportedOperationError (400) — the same way `button` rejects buttons with no
HID equivalent. As there, services() skips resolving CoreDevice for a swipe that
is going to be rejected, so it doesn't pay for tunnel setup first. Tool and schema
descriptions now state the limitation instead of implying settle works everywhere.
…ice-support

Conflict in packages/registry/src/failure-codes.ts: both sides appended to
FAILURE_CODES. Resolved as the union - main's IOS_SETTINGS_PERMISSION_FAILED /
SETTINGS_PERMISSION_UNSUPPORTED (#462) alongside this branch's CORE_DEVICE_* block.
- op_axtree: bound the audit completion wait at 10s. The event may never
  arrive (locked device, unsupported audit) and an unbounded queue get()
  wedged the strictly-serial agent forever, timing out every later op.
- CoreDeviceAgent.start(): attach a spawn 'error' handler — without one a
  spawn-layer failure (EACCES/EMFILE) is an unhandled 'error' event that
  takes down the whole tool-server — and reject the ready handshake from
  the exit/error handlers so an agent that dies at spawn fails fast
  instead of eating the full 30s timeout.
- materializeAgentScript: never execute a pre-existing tmp file without
  verifying its content (predictable shared-/tmp path was plantable by
  any local user on Linux); fall back to a private path on mismatch.
- resolvePmd3Python: reject launcher shebangs (#!/usr/bin/env python3
  'resolved' to env itself, which then treats the agent script as the
  command to exec).
- Regenerate AGENT_SCRIPT_B64 from the .py, fix the stale source-of-truth
  comment, and pin the pair with a sync test.
…ice-support

# Conflicts:
#	packages/tool-server/src/utils/ios-devices.ts
@latekvo
latekvo marked this pull request as draft July 30, 2026 19:59
@latekvo

latekvo commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

Found a way around most issues i previously believed to be dead ends. Looking into ways around the still not functional multi-touch gestures.

All installation issues should be fixed now.

Marked as ready too quickly, converting to draft for now.

ETA tuesday if i find a way for multitouch through CoreDevice, longer if it'll require xctest, long if we'll have to switch to pure xctest, but that'd be worst case.

started researching pre 27 support just now, no ETA yet. Of course we know it's trivially possible, but i'm still looking into how cleanly we can integrate it as a fallback into current architecture

latekvo and others added 5 commits July 30, 2026 22:12
… a read it cannot judge

Three defects a second review pass found on hardware, each reproduced first:

- await-screen-idle could never settle on a screen with 120 or more
  accessibility elements. Sorting the signature cancels the CoreDevice read's
  rotation only while the whole screen fits in one read; describe asks for at
  most 120 and the walk advances the VoiceOver cursor one step per read, so
  past that each poll covers a different window of the screen. A still screen
  of 121 elements took 76 polls, spent the full 15s and returned
  `settled: false` with nothing said. Reproduced with a sliding-window model
  of the device: 120 settles in 3 polls, 121 never does. It now recognises a
  truncated read and answers at once with a note naming the cause.

- The adapter stripped the VoiceOver state tokens (Dimmed / Disabled / Not
  Enabled / Selected / Not Selected) from the caption and dropped them, while
  DescribeNode carries `disabled` and `selected` and the describe hint
  promises "labels, values and traits are exact". An enabled and a disabled
  "Continue" came back byte-identical, as did a selected and an unselected
  tab. They now land on those fields.

- No element carries an `identifier`, because the audit reports no
  accessibilityIdentifier — so an identifier selector silently matches
  nothing where the same selector works on a simulator. The wire `id` is a
  per-snapshot handle, not an app identifier, so it is not a substitute:
  say so in the hint and the README instead.
A sweep at the new head found eighteen more behaviours green when broken.
The consequential ones are now pinned, each against the mutation that
exposed it:

- httpAxTree's `/api/ax-tree` path was asserted by nothing: the fetch stubs
  ignore their arguments, so pointing describe's only wire path at a route
  the sim-server does not serve left the whole suite green.
- Two `platform === "ios" && kind === "device"` guards were exercised only
  with an iPhone udid, and a physical Android phone is `kind: "device"` too.
  Dropping the platform half of gesture-custom's second-touch check silently
  takes two-contact gestures away from Android, and of await-screen-idle's
  `rotatingRead` gives Android the order-blind signature and the CoreDevice
  element ceiling. Both now have an Android control.
- The physical-UDID shape had none of its three load-bearing properties held:
  the anchors, the exact 8/16 hex runs, and case-insensitivity. The existing
  negatives fail an unanchored, loosely-quantised, uppercase-only pattern too.
- open-url's leading `^` is what makes a value starting with `-` fatal, and
  every rejected fixture lacked a colon, so the unanchored pattern rejected
  them as well.
- native-devtools' physical-iPhone backstop, while its identical ax-service
  twin was covered; and the simulator-server factory's narrowing that keeps
  the tvOS probe and the automation toggle off hardware.
- In the CoreDevice adapter: the two non-Button value-bearing roles, the
  declared trait precedence, approxFrame's clamp, and the interpolation
  denominator that puts a rect-less element inside its gap rather than
  spread across the whole list.
- coreDeviceAxTreeSchema's `rect` (the one remaining field the adapter calls
  a string method on) and `screen.h`.

Also drops `.finite()` from the schema's screen dimensions: zod 4's
`z.number()` already rejects NaN and both infinities, so it refined nothing
— which is why no mutation of it could fail.
…limit

The truncation guard refused on any full read, but a full read only means
"at least this many": a screen of exactly PHYSICAL_IOS_AX_LIMIT elements is
indistinguishable from a larger one on any single poll, and it settles
correctly, because its window IS the whole set. Re-running the sliding-window
reproduction across the boundary showed the regression - 120 elements settled
in 3 polls before the guard and refused after it.

Record that a read came back full and consult it only once the wait has
already failed, so the note explains a `settled: false` that was going to
happen anyway and never causes one. Verified across the boundary: 50, 119 and
120 settle in 3 polls with no note; 121 and 200 answer false with it.
…t have seen

A CoreDevice read returns at most 120 elements, one step further along the
VoiceOver walk each time, so on a longer screen a no-match means "outside
this window", not "gone" - describeIos says so itself on such reads ("a
'not found' here is not proof of absence"). `hidden` is the one condition a
zero-match read satisfies, so it returned success:true on the first poll for
an element that was on screen, and the caller acts on the element being
gone. Reproduced with a 200-element screen whose target sits 30 rows past
the window: hidden answered true in 1ms while `exists` found the same
element once the window reached it.

A no-match on a full read no longer satisfies `hidden` - the wait keeps
polling and, if it runs out, says why. The positive conditions are untouched:
a truncated read can delay a match, never invent one.

Also, three agent-facing docs still stated rules this feature falsified - the
udid-shape dispatch table, the "filter for platform: ios" simulator-setup
step (which now selects a connected iPhone, since it ranks ready ahead of a
shut-down simulator), and the rule's per-platform readiness list - and
propose_variant's udid description offered the preview a target the preview
whitelist drops.

Widening the ios-describe import into await-ui-element broke five test files
whose `vi.mock` of that module listed only `describeIos`; they now spread the
original, so the next export added there does not break them again.
…-simserver

# Conflicts:
#	packages/tool-server/src/blueprints/simulator-server.ts
#	packages/tool-server/src/tools/devices/boot-device.ts
#	packages/tool-server/src/tools/reinstall-app/platforms/ios.ts
#	packages/tool-server/src/utils/device-shutdown.ts
#	packages/tool-server/src/utils/ios-devices.ts
The CoreDevice touchscreen was taken to be single-contact, so gesture-pinch
and gesture-rotate were gated off hardware entirely and gesture-custom rejected
any event carrying x2/y2.

It takes two. Byte 1 of the touchscreen's report is a contact identifier
(radon: `ios_device::hid`), and reports carrying different ids are tracked as
different fingers. Verified on an iPhone 15 / iOS 27: a two-finger pinch driven
end to end through the tool-server zooms Apple Maps in and back out, against a
before-shot proving the screen was static.

gesture-rotate stays gated, and now says why: two contacts do arrive, but iOS
does not read a rotation out of them. The identical arc — same centre, radii and
frame rate — turns the map on a simulator and raises its compass, while on
hardware it only pans, across sweeps from 90 to 150 degrees and durations from
0.5s to 2s. Better gated than returning `rotated: true` over a screen that did
not turn.

Also drops the per-request x2/y2 rejection in gesture-custom, along with the
no-service special case that existed only to avoid spawning a session for a
request that was going to be rejected.
filip131311 added a commit that referenced this pull request Aug 3, 2026
…ounts are zero (#649)

Fixes #644.

## Before

```
$ argent flow run echo-only          # .argent/flows/echo-only.yaml: two `echo` steps
[Tool:flow-execute] No booted device found. Pass a device id or platform explicitly.
Available devices: 18BE573F-… (ios, Shutdown), <70 more>
```

and when a device *was* booted:

```
PASS on 18BE573F-… — 0 passed, 0 failed, 0 errored, 0 skipped
```

## After

```
$ argent flow run echo-only          # nothing booted
Flow "echo-only"
  › first step
  › second step

PASS — 0 passed, 0 failed, 0 errored, 0 skipped (no test steps)
```

## Symptom 1 — the device

Whether a device is needed is decided from the flow's own steps, before
resolution.

Classification is **per step kind and defaults to needing one**, so a
kind added later inherits
today's behaviour instead of silently running against no device — and
the `never` binding makes
leaving a new kind unclassified a compile error, backed by a test keyed
on the step union so it
fails to build rather than to run.

Two classifications worth calling out:

- **`when` always needs a device**, whatever its body contains, because
the guard reads one itself
(its platform, or its view tree). A `when: {platform: ios}` block
containing only narration still
  resolves.
- **`run` needs one without the fragment being read here.** The named
flow is resolved at run time;
resolving it a second time would duplicate that lookup and could
disagree with it if the file
changed in between. A deliberate over-approximation: composing a
narration-only fragment still
  resolves a device. Stated in the code and in the tool description.

For `tool:` steps the tool's own schema decides, via a superset of the
keys the runner injects —
`device` is included because a nested `flow-execute` step drives a
device without being handed the
run's own. A tool that declares no input at all
(`stop-all-simulator-servers`) is device-free, and
reading its absent schema must not throw — it previously would have.

`ExecState.device` is nullable, which is the safety mechanism rather
than just typing: the compiler
enumerates every site that acts on a device, each of which now says so
via one `deviceEnv()`
narrowing. `flow-actions.ts`, `flow-visual.ts` and `ActionEnv` are
untouched. The executor also
re-checks each step against the **same predicate**, so if the scan and
the executor ever disagree
the step reports it instead of failing obscurely deeper in.

A run that resolved no device reports `device: ""`, and neither renderer
claims it ran somewhere.

## Symptom 2 — the counters

Reproducing this showed the issue's framing is slightly off: `--json`
doesn't only disagree with the
text, it disagrees with **itself** — counters all zero while its own
`steps[]` marks both steps
`"pass"`. So the CLI text was a faithful rendering of the server's
counters.

**The counters are right and stay as they are.** Narration is
deliberately excluded, and it should
be: after a hard stop every remaining step *including narration* is
reported skipped, so counting it
would inflate `skipped` on every real failing run — a worse and far more
common lie than the vacuous
one being reported here. It would also contradict the JUnit model in
#578, where echo is
`<system-out>` and not a `<testcase>`.

What was missing was saying *why* zero. Both the CLI and the MCP
renderer now append
`(no test steps)`, and only on a pass — a cancelled run can be a FAIL
with all four counters zero,
where "no test steps" would read as though the failure had no cause.

No wire format changes: `steps[].kind === "echo"` already distinguishes
counted from uncounted steps,
so the JSON is self-describing without widening the status enum.

## Conflicts, for whoever lands second

This is a hot area — seven open PRs touch `flow-run.ts`.

- **#578** rewrites the CLI summary line I also change (one hunk, one
7-line function, semantically
independent — resolution is keep-both), and its
`attachFailureDiagnostics` dereferences
`env.device.platform`. **It will need a null-device guard**, or every
failure in a device-less flow
silently degrades to `read-failed` diagnostics. Happy to rebase onto it
rather than the reverse.
- **#585** rewrites `resolveRunDevice` at the same seam and edits the
same description sentence.
- **#538** rewrites `flow-device.ts`. Worth noting it makes
`resolveFlowDevice` reject a physical
iPhone for flows — skipping resolution means a narration-only flow now
passes on a
physical-iPhone-only host, which I think is desirable but is a second
behaviour change on this seam.

All new tests are in new files, since every existing flow test file in
the blast radius is rewritten
by an open PR.

## Behaviour changes a reviewer should weigh

- A device-free flow no longer prints `PASS on <id>` **even when a
device is booted**. That is the
point: otherwise an identical flow reports differently in CI (no
simulator) than on a laptop with
  one open. An explicit `--device` is still honoured and still named.
- `--platform` is ignored for a device-free flow, where it previously
errored with nothing booted.
- A 0.18.1 CLI against a newer server would print `Flow "x" on ` —
cosmetic version skew.

## Verification

Live against the built CLI: device-less flow with nothing booted (the
reported bug) and with a device
booted (identical output, no ambient attribution); `--json` showing
`device: ""`; explicit `--device`
still naming it; a mixed flow and a composed flow both still demanding a
device; a mixed flow with a
device booted still reporting `PASS on <id> — 1 passed`.

24 new tests. tool-server 3103 passed, CLI 284 passed, MCP 78 passed —
no existing test modified.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

> **#697 is stacked on this branch** — it conflicts with this one in
`flow-run.ts`'s import block, so it was rebased on top. Merge this
first.
latekvo added 7 commits August 4, 2026 17:21
…iPhone"

This reverts commits 7a1be55 and fd0bd3a.

The premise was that the `mainTouchscreen` report's contact-id byte gives a
physical iPhone two fingers. It does not. The byte is an identifier and the
device accepts reports addressed to different ids, but iOS reads one finger
off them regardless. Re-measured on an iPhone 15 / iOS 27 against an iOS
18.6 simulator receiving byte-identical input, on Apple Maps:

| gesture                      | simulator          | iPhone                      |
| ---------------------------- | ------------------ | --------------------------- |
| two-finger tap               | zooms out one step | nothing                     |
| two-finger vertical drag     | enters 3D          | pans                        |
| symmetric pinch 0.35 -> 0.05 | zooms out          | pans by one contact's delta |
| two-finger hold 600ms        | nothing            | drops a pin (a long press)  |

The pin is the clearest of the four: a long press only fires for a single
finger, so during a two-contact hold the device had one finger on the glass.
The pinch is the most quantitative: the map translated by exactly one
contact's displacement at unchanged zoom.

The original measurement read a pan of that kind as a zoom. A pinch that far
apart moves the map a long way, which looks like something happened.

So `gesture-pinch` and `gesture-rotate` go back to `apple.device: false` and
`gesture-custom` rejects its two-point form on hardware again — a shipped
`pinched: true` over a screen that only panned is the outcome worth
avoiding. Needs radon a7f4470b, which reverts the sim-server half.
…tions

settings-permissions: name what was actually checked - devicectl carries no
privacy verb in its command tree, and the configuration-profile route needs a
CMS-signed profile, so it folds into the signing requirement.

paste: drop it. The claim that its capability gate is what stops a paste from
silently going nowhere was wrong - the tool is not in setup-registry.ts on any
platform, so the gate never runs and nothing can call it. It is not a
physical-iPhone limitation.
…gated

The stated reason - that xctrace can only enumerate processes it is entitled
to profile, leaving nothing to attach to - does not hold. On a host with zero
codesigning identities, `xctrace record --device <ecid> --attach <pid>` against
a tethered iPhone 15 on iOS 27 produced a 37MB Time Profiler trace of
SpringBoard with symbolicated user-space frames.

The gate is right, but for the reason the capability comment already gives:
the iOS capture path resolves its target through `simctl spawn` and
`simctl listapps`, which are simulator-only. That is an unimplemented device
path, not a platform wall, and the README should not claim otherwise.
…hat was measured

xctrace: the device-profiling note read as unconditional. It was measured on
Xcode 16.4, and native-profiler's own iOS capture path documents a --device
recording-start deadlock for Xcode 26.4 and later, so an unqualified claim
points a reader on a newer Xcode at a known bug. Name the version, and record
that attach works by pid but not by name.

Frames: "Apple exposes no per-element geometry app-free on a physical device"
asserts more than has been established - #702 found no frame attribute in the
published inspector set, but the hit-test selector's encoding is unresolved and
the audit path does carry ElementRectValue_v1. Say what the read carries.
Both the README and the describe hint an agent reads on every physical-iOS
call said the audit reports no accessibilityIdentifier. It does report them:
_AXHierarchyElementsAttribute carries AccessibilityIdentifier_v1 with real
values, and the read simply does not parse them (#702). An agent told the
platform withholds identifiers stops looking; one told this read does not
surface them knows the selector is unavailable today without inferring a limit
that is not there. Behaviour is unchanged - an identifier selector still
matches nothing.

Same correction to the hint's frames clause: the read carries no geometry,
which is what was measured, rather than the device exposing none.
…eometry comments

Three comments said Apple exposes no per-element geometry app-free on hardware.
What was established is narrower: the inspector this read goes through publishes
no frame attribute. The hit-test selector's argument encoding is unresolved and
the audit path does carry ElementRectValue_v1 (#702), so the broader claim would
tell the next reader the question is closed when it is open.

No behaviour change - frames stay synthesised and the wire contract's rect stays
unsent.

@latekvo latekvo left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Diplomat]: Static review of this PR: read the full diff, then each changed file in whole together with its callers and its nearest sibling implementation (other platform, other call site, other arm of the same conditional), and checked every added or changed test against the question of whether it would still pass with the production hunk reverted. Pre-PR behaviour was read from the merge base (git show <merge-base>:<path>) rather than inferred.

Two reviewers read this PR on different lenses. Each observation below survived an adversarial refutation check against the code; anything the check could not confirm was dropped.

Nothing here was reproduced on a device — this was a static read.

Comment thread packages/skills/rules/argent.md Resolved
latekvo and others added 2 commits August 17, 2026 12:44
Neither interface is imported anywhere: list-devices declares its own IosDevice
now that a physical entry carries kind and productType, so exporting these only
widened the surface knip's dead-code ceiling counts.
@latekvo
latekvo marked this pull request as ready for review August 17, 2026 11:26
…-simserver

Two conflicts, both where main edited lines this branch had also touched:

- boot-device.ts: main moved the headless-boot helpers into
  utils/no-window-env.ts (androidHeadlessFromEnv / iosHeadlessFromEnv) in the
  same import block this branch extended with isPhysicalIosUdid and
  simulatorServerRef. Kept both sides. The physical-iPhone branch in bootIos
  returns before the Simulator.app GUI attach, so iosHeadlessFromEnv is never
  consulted for a hardware udid.

- describe/platforms/ios/index.ts: main reworded the comment above
  NON_INJECTABLE_HINT ("can never load argent's injected dylib" -> "cannot be
  relied on to load argent's dylib", matching #560's measured verdict) where
  this branch had inserted the physical-iOS hint constants directly above it.
  Kept the new constants and took main's wording.
…-simserver

# Conflicts:
#	packages/tool-server/src/blueprints/ax-service.ts
#	packages/tool-server/src/blueprints/simulator-server.ts
#	packages/tool-server/src/preview.ts
#	packages/tool-server/src/tools/await-screen-idle/index.ts
#	packages/tool-server/src/tools/await-ui-element/index.ts
#	packages/tool-server/src/tools/describe/contract.ts
#	packages/tool-server/src/tools/describe/platforms/ios/index.ts
#	packages/tool-server/src/tools/devices/list-devices.ts
#	packages/tool-server/src/tools/paste/index.ts
#	packages/tool-server/src/tools/profiler/combined/profiler-combined-report.ts
#	packages/tool-server/src/tools/profiler/query/profiler-stack-query.ts
#	packages/tool-server/src/tools/run-sequence/index.ts
#	packages/tool-server/src/tools/screenshot/index.ts
#	packages/tool-server/src/utils/device-info.ts
#	packages/tool-server/src/utils/device-shutdown.ts
#	packages/tool-server/src/utils/ios-devices.ts
#	packages/tool-server/src/utils/simulator-client.ts
- list-devices: keep the promised deviceSet tag on iOS simulator entries
- simulator-server: classify spawn failures on 'close', not 'exit', so
  stderr delivered after exit is not lost to the race
- skills/rules/argent.md: carve the physical-iPhone screenshot-first
  workflow out of the tapping rule that banned it
- await-screen-idle test: make the Android control fail when the
  rotatingRead platform narrowing is dropped
- followups test: add settings-permissions to the pinned simulator-only
  set and defer exhaustiveness to the registry-derived sweep
- paste: correct the apple.device rationale and the sweep's note about
  why paste is defined outside createRegistry
@latekvo

latekvo commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

Closing for now, will be relevant after the xctest based approach is merged.

@latekvo latekvo closed this Aug 31, 2026
hubgan added a commit that referenced this pull request Oct 30, 2026
…iPhone (#1664)

## Summary

- Add a `touchPaths` command to the iOS device runner: one timed
waypoint path per finger (1 to 5), played at once as a single event
record through XCTest's event synthesizer.
- `gesture-pinch`, `gesture-rotate` and `gesture-custom` now run on a
physical iPhone through it. e2e `pinch`/`rotate` steps no longer fail
upfront there.
- A plain Down/Up `gesture-custom` (press-hold, drag, long-press pickup)
stays on `XCUICoordinate`, whose hold semantics a pickup needs. Only a
second finger or `Move` waypoints take the new path.
- Update the docs, rules and skills that said physical iPhones have no
two-finger gestures, and drop the same line from `describe`'s header.

## Why

On hardware the runner could only draw one straight press-then-drag, so
an agent testing an app with a pinch-to-zoom view, a map, a canvas, or
any custom multi-touch handler had to give up or drive the app's zoom
buttons. The docs and tool descriptions say "XCTest exposes no
two-finger coordinate API". That is true of the public `XCUICoordinate`
API, but XCTest builds its own touches from `XCPointerEventPath` and
`XCSynthesizedEventRecord`, and WebDriverAgent has used them for W3C
multi-finger actions for years.

## How it works

- `ArgentTouchSynthesizer` (Objective-C) looks both classes up with
`NSClassFromString` and checks every selector with
`instancesRespondToSelector:` before use. Nothing links against a
private symbol.
- If an Xcode renames or drops one, `touchPaths` answers
`UNSUPPORTED_OPERATION` with the name of what is missing. Every other
command keeps working, and single-finger gestures are unaffected.
- The runner is a test bundle installed on the developer's own phone and
never ships through the App Store, so the private API is a maintenance
risk but not a review risk. I would like your view on whether that trade
is acceptable. There is no public route to a second finger.
- Wire shape: `fingers: [{ points: [{ x, y, tMs }] }]`, absolute points
in the app's space like `tap` and `drag`. A finger whose first waypoint
is later than 0 touches down late, which is how a second finger joins a
gesture in progress. Documented in `PROTOCOL.md`.
- Validation is on the runner: 1 to 5 fingers, finite values, times
within 0 to 30 000 ms and non-decreasing per path.
- Pinch and rotate geometry moved from the tools into `gesture-utils` so
Harmony, the simulator loops and the iPhone path share one
implementation. Frames are 16 ms apart, the cadence the simulator and
emulator paths already use.
- The event record is built with
`initWithName:displayID:interfaceOrientation:` when XCTest and the main
screen offer a display id, and with the older
`initWithName:interfaceOrientation:` otherwise. Either one passes the
availability check, so an Xcode that keeps only one of them still works.
- Only an app drawn in **portrait** is accepted. The check reads the
app's own interface orientation (`XCUIApplication`'s
`interfaceOrientation`), not the way the phone is held, so a
portrait-only app on a phone turned sideways is not refused. An app
drawn in landscape is refused because I have not checked landscape on
hardware, and a touch that lands somewhere else without an error is
worse than a refusal. An orientation the runner cannot read is refused
too, rather than guessed. Lifting this needs a point mapping into the
panel's portrait space: callstack/agent-device#3838 has one, and I can
add it once someone has verified landscape on a phone.

## Verified on hardware

iPhone 12, iOS 27.0.1, Xcode 27.1, driven through the tool-server built
from this branch with `ARGENT_IOS_RUNNER_PROJECT` pointing at it, Safari
on a Wikipedia article. There were two runs, because the third commit
changed which record initializer is used.

**First two commits** (`initWithName:interfaceOrientation:`):

| Call | Result |
| --- | --- |
| `gesture-pinch` 0.1 to 0.6 over 400 ms | page zoomed about 3x |
| `gesture-pinch` 0.6 to 0.1 | zoomed back out |
| `gesture-rotate` | returns `rotated: true` (Safari has no rotate
handler, so there is nothing visible) |
| `gesture-custom`, one finger, 3 `Move` waypoints | page scrolled |
| `gesture-custom`, second finger joining late | page zoomed further |
| `gesture-custom`, plain Down/Up | still takes the `drag` path |
| `gesture-custom`, train starting with `Move` | refused before touching
the device |

**Third commit** (`initWithName:displayID:interfaceOrientation:`,
display id 1, orientation read as portrait). A temporary log line,
removed before committing, confirmed both values. I then re-ran on the
committed code:

| Call | Result |
| --- | --- |
| `gesture-custom`, four-event train at the off-centre "Bell Labs" link
(takes `touchPaths`, not `drag`) | opened the Bell Labs article, so the
touch landed where it was sent |
| `gesture-pinch` out, then in | zoomed, then back |
| `gesture-rotate` | returns `rotated: true` |
| `gesture-custom`, train starting with `Move` | refused before touching
the device |

I confirmed the synthesizer in a standalone XCUITest before wiring it
into the runner.

**Not verified:** other Xcode or iOS versions, iPads (not supported by
the runner), landscape (refused), a portrait-only app on a phone held
sideways (accepted by the check but not tried), any app other than
Safari, and timing fidelity. In the standalone probe a 0.6 s script took
0.96 s end to end, so expect gestures to run somewhat longer than their
nominal duration.

## Tests

- `ios-device-touch-paths.test.ts` (new): exact waypoints for pinch
(spread, odd duration, angle plus drifting centre, off-screen clamp),
rotate, and trains (delay spacing, late and early second finger,
refusals), plus the `touchPaths` wire shape and the pinch/rotate tools
on a device.
- `gesture-custom.test.ts`: the "rejects what XCTest cannot replay"
cases are replaced with the new behaviour and the remaining refusals.
- `flow-pinch` / `flow-rotate` / `describe-format-tree` / capability
ratchet updated for the new behaviour.
- `npm run build`, `npm run knip` and eslint on the changed files pass.
The full tool-server suite has 3 failures that are identical on an
untouched `main` in my environment (two real-ffmpeg letterbox tests and
the `e2e-tool-aliases` listing test), so I left them alone.
- There is no unit-test harness for the Swift side, so the runner
changes are covered by the hardware runs above.

## Related

- #944 (merged) listed this as a known gap: "gesture-pinch,
gesture-rotate: no two-finger coordinate API on hardware", and made the
e2e pinch/rotate steps fail upfront. This PR lifts that limit and
removes the gate. I found no open issue or PR for it.
- #538 (closed) wanted multi-touch on hardware through CoreDevice and
noted it might need XCTest instead. This is the XCTest route.
- callstack/agent-device#3838 added the same capability to
agent-device's XCTest runner through the same private classes, verified
on an iPhone 12. The third commit follows its choice of record
initializer and its use of the app's interface orientation.
- #489 (draft, no activity since 10 Jul) proposed pinch and rotate
through WebDriverAgent, before the XCUITest runner existed. This is the
runner-native equivalent.
- #1528 and #1624 also edit `ArgentRunnerSession+Gestures.swift` and
`RunnerProtocol.swift` (#1528 also `PROTOCOL.md` and
`runner-commands.ts`). The features are independent, but expect small
textual conflicts in whichever merges second. #1624 changes how window
coordinates are mapped on foldables, so it is worth a look alongside
this PR's portrait-only rule.
- Out of scope: Android. Physical-Android two-finger gestures are
blocked by #1628 / #1645, which live in the closed-source
`simulator-server`.

---------

Co-authored-by: Hubert Gancarczyk <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants