fix: add explicit native taps for physical iOS - #1528

…iPhone (#1664)

## Summary

- Add a `touchPaths` command to the iOS device runner: one timed
waypoint path per finger (1 to 5), played at once as a single event
record through XCTest's event synthesizer.
- `gesture-pinch`, `gesture-rotate` and `gesture-custom` now run on a
physical iPhone through it. e2e `pinch`/`rotate` steps no longer fail
upfront there.
- A plain Down/Up `gesture-custom` (press-hold, drag, long-press pickup)
stays on `XCUICoordinate`, whose hold semantics a pickup needs. Only a
second finger or `Move` waypoints take the new path.
- Update the docs, rules and skills that said physical iPhones have no
two-finger gestures, and drop the same line from `describe`'s header.

## Why

On hardware the runner could only draw one straight press-then-drag, so
an agent testing an app with a pinch-to-zoom view, a map, a canvas, or
any custom multi-touch handler had to give up or drive the app's zoom
buttons. The docs and tool descriptions say "XCTest exposes no
two-finger coordinate API". That is true of the public `XCUICoordinate`
API, but XCTest builds its own touches from `XCPointerEventPath` and
`XCSynthesizedEventRecord`, and WebDriverAgent has used them for W3C
multi-finger actions for years.

## How it works

- `ArgentTouchSynthesizer` (Objective-C) looks both classes up with
`NSClassFromString` and checks every selector with
`instancesRespondToSelector:` before use. Nothing links against a
private symbol.
- If an Xcode renames or drops one, `touchPaths` answers
`UNSUPPORTED_OPERATION` with the name of what is missing. Every other
command keeps working, and single-finger gestures are unaffected.
- The runner is a test bundle installed on the developer's own phone and
never ships through the App Store, so the private API is a maintenance
risk but not a review risk. I would like your view on whether that trade
is acceptable. There is no public route to a second finger.
- Wire shape: `fingers: [{ points: [{ x, y, tMs }] }]`, absolute points
in the app's space like `tap` and `drag`. A finger whose first waypoint
is later than 0 touches down late, which is how a second finger joins a
gesture in progress. Documented in `PROTOCOL.md`.
- Validation is on the runner: 1 to 5 fingers, finite values, times
within 0 to 30 000 ms and non-decreasing per path.
- Pinch and rotate geometry moved from the tools into `gesture-utils` so
Harmony, the simulator loops and the iPhone path share one
implementation. Frames are 16 ms apart, the cadence the simulator and
emulator paths already use.
- The event record is built with
`initWithName:displayID:interfaceOrientation:` when XCTest and the main
screen offer a display id, and with the older
`initWithName:interfaceOrientation:` otherwise. Either one passes the
availability check, so an Xcode that keeps only one of them still works.
- Only an app drawn in **portrait** is accepted. The check reads the
app's own interface orientation (`XCUIApplication`'s
`interfaceOrientation`), not the way the phone is held, so a
portrait-only app on a phone turned sideways is not refused. An app
drawn in landscape is refused because I have not checked landscape on
hardware, and a touch that lands somewhere else without an error is
worse than a refusal. An orientation the runner cannot read is refused
too, rather than guessed. Lifting this needs a point mapping into the
panel's portrait space: callstack/agent-device#3838 has one, and I can
add it once someone has verified landscape on a phone.

## Verified on hardware

iPhone 12, iOS 27.0.1, Xcode 27.1, driven through the tool-server built
from this branch with `ARGENT_IOS_RUNNER_PROJECT` pointing at it, Safari
on a Wikipedia article. There were two runs, because the third commit
changed which record initializer is used.

**First two commits** (`initWithName:interfaceOrientation:`):

| Call | Result |
| --- | --- |
| `gesture-pinch` 0.1 to 0.6 over 400 ms | page zoomed about 3x |
| `gesture-pinch` 0.6 to 0.1 | zoomed back out |
| `gesture-rotate` | returns `rotated: true` (Safari has no rotate
handler, so there is nothing visible) |
| `gesture-custom`, one finger, 3 `Move` waypoints | page scrolled |
| `gesture-custom`, second finger joining late | page zoomed further |
| `gesture-custom`, plain Down/Up | still takes the `drag` path |
| `gesture-custom`, train starting with `Move` | refused before touching
the device |

**Third commit** (`initWithName:displayID:interfaceOrientation:`,
display id 1, orientation read as portrait). A temporary log line,
removed before committing, confirmed both values. I then re-ran on the
committed code:

| Call | Result |
| --- | --- |
| `gesture-custom`, four-event train at the off-centre "Bell Labs" link
(takes `touchPaths`, not `drag`) | opened the Bell Labs article, so the
touch landed where it was sent |
| `gesture-pinch` out, then in | zoomed, then back |
| `gesture-rotate` | returns `rotated: true` |
| `gesture-custom`, train starting with `Move` | refused before touching
the device |

I confirmed the synthesizer in a standalone XCUITest before wiring it
into the runner.

**Not verified:** other Xcode or iOS versions, iPads (not supported by
the runner), landscape (refused), a portrait-only app on a phone held
sideways (accepted by the check but not tried), any app other than
Safari, and timing fidelity. In the standalone probe a 0.6 s script took
0.96 s end to end, so expect gestures to run somewhat longer than their
nominal duration.

## Tests

- `ios-device-touch-paths.test.ts` (new): exact waypoints for pinch
(spread, odd duration, angle plus drifting centre, off-screen clamp),
rotate, and trains (delay spacing, late and early second finger,
refusals), plus the `touchPaths` wire shape and the pinch/rotate tools
on a device.
- `gesture-custom.test.ts`: the "rejects what XCTest cannot replay"
cases are replaced with the new behaviour and the remaining refusals.
- `flow-pinch` / `flow-rotate` / `describe-format-tree` / capability
ratchet updated for the new behaviour.
- `npm run build`, `npm run knip` and eslint on the changed files pass.
The full tool-server suite has 3 failures that are identical on an
untouched `main` in my environment (two real-ffmpeg letterbox tests and
the `e2e-tool-aliases` listing test), so I left them alone.
- There is no unit-test harness for the Swift side, so the runner
changes are covered by the hardware runs above.

## Related

- #944 (merged) listed this as a known gap: "gesture-pinch,
gesture-rotate: no two-finger coordinate API on hardware", and made the
e2e pinch/rotate steps fail upfront. This PR lifts that limit and
removes the gate. I found no open issue or PR for it.
- #538 (closed) wanted multi-touch on hardware through CoreDevice and
noted it might need XCTest instead. This is the XCTest route.
- callstack/agent-device#3838 added the same capability to
agent-device's XCTest runner through the same private classes, verified
on an iPhone 12. The third commit follows its choice of record
initializer and its use of the app's interface orientation.
- #489 (draft, no activity since 10 Jul) proposed pinch and rotate
through WebDriverAgent, before the XCUITest runner existed. This is the
runner-native equivalent.
- #1528 and #1624 also edit `ArgentRunnerSession+Gestures.swift` and
`RunnerProtocol.swift` (#1528 also `PROTOCOL.md` and
`runner-commands.ts`). The features are independent, but expect small
textual conflicts in whichever merges second. #1624 changes how window
coordinates are mapped on foldables, so it is worth a look alongside
this PR's portrait-only rule.
- Out of scope: Android. Physical-Android two-finger gestures are
blocked by #1628 / #1645, which live in the closed-source
`simulator-server`.

---------

Co-authored-by: Hubert Gancarczyk <[email protected]>

…at never go idle (#1722)

On a physical iPhone, a gesture on a screen that never goes idle
(Bluesky's profile: expo-blur keeps a paused `UIViewPropertyAnimator`)
took 60-120 s and then failed with `RUNNER_BUSY`. Two things caused it:

- **Slow:** XCTest waits up to 60 s before and after every event for the
app to go idle.
- **Repeated:** the MCP adapter aborts a call after 30 s and sends it
again. The tool-server keeps running each aborted call, so one
`gesture-tap` landed 2-5 times.

**Fix:**
- The runner skips XCTest's idle waits and instead waits at most 2 s for
the app to settle before each event.
- `gesture-tap`, `gesture-swipe`, `gesture-custom`, `gesture-pinch`,
`gesture-rotate` and `button` are `longRunning`, as `keyboard` already
is.

| Stalled screen, one MCP `gesture-tap` | main | this PR |
|---|---|---|
| time | 135 s, then `RUNNER_BUSY` | ~5 s, ok |
| physical taps | 2 | 1 |
| text entry (focus / type) | 75 s / 55 s, both timed out | 2.3 s / 3.4
s, text landed |

> [!NOTE]
> **This uses private XCTest API.** There's no screenshotting or
sleep-polling. `ArgentRunnerSession+Quiescence.swift`:
> 1. Calls the same check XCTest runs on its own (`applicationImpl` →
`currentProcess` → `_initiateQuiescenceChecksIncludingAnimationsIdle:`
and `_makeQuiescenceExpectation`), but waits on it for at most 2 s
instead of 60 s.
> 2. Sends the event inside `-[XCUIApplication
_performWithInteractionOptions:block:]` with options `0b11` (skip the
pre-event and post-event waits). This is the same SPI agent-device uses.
>
> Selectors are resolved with `NSSelectorFromString` and called through
`unsafeBitCast` IMPs, and each one is checked with `responds(to:)`
first. If an Xcode drops one, the gesture falls back to XCTest's normal
60 s waits rather than crashing.
>
> **Cost:** on a screen that never goes idle, every gesture now waits
the full 2 s. On main it waited 60-120 s. Skipping the wait entirely
(0.4 s) lost taps sent right after a navigation (0/5 landed).

Docs: no docs-site change. `longRunning` is internal, and the runner's
PROTOCOL.md/README are updated.

<details>
<summary>Repro and verification</summary>

No physical iPhone was used. The real runner ran on an iOS 18.5
simulator, with stub `xcrun`/`xcodebuild` and a fake usbmuxd making the
tool-server treat it as a phone. The tool-server, MCP adapter and runner
were the real code. The fixture app stalls XCTest with either a paused
`UIViewPropertyAnimator` on a visual-effect view or a repeating
`UIView.animate`.

Direct to the runner:

| Case | main | this PR |
|---|---|---|
| Stalled screen, 3 taps | 60.5 s ok; 75.1 s `COMMAND_TIMED_OUT`; then
`RUNNER_BUSY` | 0.41 s, then 2.32-2.35 s each; 6/6 landed (both stall
types) |
| Stalled screen, focus / type / delete / return | 75 / 55 / 30 / 30 s,
all timed out | 2.34 / 3.36 / 3.32 / 2.42 s, all ok |
| Stalled screen, doubleTap / longPress / drag | - | 2.48 / 3.10 / 2.80
s, all landed |
| Normal screen: open modal, wait 100 ms, close (5 runs) | 5/5, close
took 1.0-1.1 s | 5/5, close took 0.75-0.82 s |
| `touchPaths` pinch (from #1664), stalled vs normal | - | ~0.74 s on
both, 3/3 registered (synthesized directly, never waited for idle) |

Through MCP, first tap of a session with the runner build padded to 45
s: main gave 2 tool calls and 2 taps (the app saw a double tap); this PR
gives 1 call and 1 tap.

Local checks: `npm run build`, `typecheck:tests`, prettier and eslint
pass. vitest passed 826 tests across the gesture, button, keyboard,
http, ios-device, ios-physical and run-sequence suites. The runner
builds for a real device (`build-for-testing generic/platform=iOS`).
`ios-device-gestures-long-running.test.ts` fails when any single
`longRunning` flag is removed.

Not fixed here:
- A first runner build longer than 300 s is still replayed (#1632).
- Bringing a backgrounded app to the foreground still uses XCTest's
normal waits, capped at about 15 s.
- #1528's `element.tap()` would need the same `skippingQuiescence`
wrapper.
</details>