Skip to content

feat(ios-device): pinch, rotate and multi-finger paths on a physical iPhone - #1664

Merged
hubgan merged 4 commits into
software-mansion:mainfrom
camchis:feat/ios-device-multitouch
Oct 30, 2026
Merged

hubgan merged 4 commits into
software-mansion:mainfrom
camchis:feat/ios-device-multitouch

Conversation

@camchis

@camchis camchis commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add a touchPaths command to the iOS device runner: one timed waypoint path per finger (1 to 5), played at once as a single event record through XCTest's event synthesizer.
  • gesture-pinch, gesture-rotate and gesture-custom now run on a physical iPhone through it. e2e pinch/rotate steps no longer fail upfront there.
  • A plain Down/Up gesture-custom (press-hold, drag, long-press pickup) stays on XCUICoordinate, whose hold semantics a pickup needs. Only a second finger or Move waypoints take the new path.
  • Update the docs, rules and skills that said physical iPhones have no two-finger gestures, and drop the same line from describe's header.

Why

On hardware the runner could only draw one straight press-then-drag, so an agent testing an app with a pinch-to-zoom view, a map, a canvas, or any custom multi-touch handler had to give up or drive the app's zoom buttons. The docs and tool descriptions say "XCTest exposes no two-finger coordinate API". That is true of the public XCUICoordinate API, but XCTest builds its own touches from XCPointerEventPath and XCSynthesizedEventRecord, and WebDriverAgent has used them for W3C multi-finger actions for years.

How it works

  • ArgentTouchSynthesizer (Objective-C) looks both classes up with NSClassFromString and checks every selector with instancesRespondToSelector: before use. Nothing links against a private symbol.
  • If an Xcode renames or drops one, touchPaths answers UNSUPPORTED_OPERATION with the name of what is missing. Every other command keeps working, and single-finger gestures are unaffected.
  • The runner is a test bundle installed on the developer's own phone and never ships through the App Store, so the private API is a maintenance risk but not a review risk. I would like your view on whether that trade is acceptable. There is no public route to a second finger.
  • Wire shape: fingers: [{ points: [{ x, y, tMs }] }], absolute points in the app's space like tap and drag. A finger whose first waypoint is later than 0 touches down late, which is how a second finger joins a gesture in progress. Documented in PROTOCOL.md.
  • Validation is on the runner: 1 to 5 fingers, finite values, times within 0 to 30 000 ms and non-decreasing per path.
  • Pinch and rotate geometry moved from the tools into gesture-utils so Harmony, the simulator loops and the iPhone path share one implementation. Frames are 16 ms apart, the cadence the simulator and emulator paths already use.
  • The event record is built with initWithName:displayID:interfaceOrientation: when XCTest and the main screen offer a display id, and with the older initWithName:interfaceOrientation: otherwise. Either one passes the availability check, so an Xcode that keeps only one of them still works.
  • Only an app drawn in portrait is accepted. The check reads the app's own interface orientation (XCUIApplication's interfaceOrientation), not the way the phone is held, so a portrait-only app on a phone turned sideways is not refused. An app drawn in landscape is refused because I have not checked landscape on hardware, and a touch that lands somewhere else without an error is worse than a refusal. An orientation the runner cannot read is refused too, rather than guessed. Lifting this needs a point mapping into the panel's portrait space: feat(apple): admit two-finger gestures on physical iOS devices callstack/agent-device#3838 has one, and I can add it once someone has verified landscape on a phone.

Verified on hardware

iPhone 12, iOS 27.0.1, Xcode 27.1, driven through the tool-server built from this branch with ARGENT_IOS_RUNNER_PROJECT pointing at it, Safari on a Wikipedia article. There were two runs, because the third commit changed which record initializer is used.

First two commits (initWithName:interfaceOrientation:):

Call Result
gesture-pinch 0.1 to 0.6 over 400 ms page zoomed about 3x
gesture-pinch 0.6 to 0.1 zoomed back out
gesture-rotate returns rotated: true (Safari has no rotate handler, so there is nothing visible)
gesture-custom, one finger, 3 Move waypoints page scrolled
gesture-custom, second finger joining late page zoomed further
gesture-custom, plain Down/Up still takes the drag path
gesture-custom, train starting with Move refused before touching the device

Third commit (initWithName:displayID:interfaceOrientation:, display id 1, orientation read as portrait). A temporary log line, removed before committing, confirmed both values. I then re-ran on the committed code:

Call Result
gesture-custom, four-event train at the off-centre "Bell Labs" link (takes touchPaths, not drag) opened the Bell Labs article, so the touch landed where it was sent
gesture-pinch out, then in zoomed, then back
gesture-rotate returns rotated: true
gesture-custom, train starting with Move refused before touching the device

I confirmed the synthesizer in a standalone XCUITest before wiring it into the runner.

Not verified: other Xcode or iOS versions, iPads (not supported by the runner), landscape (refused), a portrait-only app on a phone held sideways (accepted by the check but not tried), any app other than Safari, and timing fidelity. In the standalone probe a 0.6 s script took 0.96 s end to end, so expect gestures to run somewhat longer than their nominal duration.

Tests

  • ios-device-touch-paths.test.ts (new): exact waypoints for pinch (spread, odd duration, angle plus drifting centre, off-screen clamp), rotate, and trains (delay spacing, late and early second finger, refusals), plus the touchPaths wire shape and the pinch/rotate tools on a device.
  • gesture-custom.test.ts: the "rejects what XCTest cannot replay" cases are replaced with the new behaviour and the remaining refusals.
  • flow-pinch / flow-rotate / describe-format-tree / capability ratchet updated for the new behaviour.
  • npm run build, npm run knip and eslint on the changed files pass. The full tool-server suite has 3 failures that are identical on an untouched main in my environment (two real-ffmpeg letterbox tests and the e2e-tool-aliases listing test), so I left them alone.
  • There is no unit-test harness for the Swift side, so the runner changes are covered by the hardware runs above.

Related

camchis and others added 4 commits October 28, 2026 22:16
…stures

XCUICoordinate draws one straight press-then-drag, so the runner could not
play a pinch, a rotation or a path with waypoints. Add `touchPaths`: one timed
waypoint path per finger (1 to 5), played at once as a single event record
through XCTest's own XCPointerEventPath / XCSynthesizedEventRecord.

Those classes are private, so ArgentTouchSynthesizer resolves them by name at
runtime and checks every selector first. Nothing links against a private
symbol, and an Xcode that drops one makes the command answer
UNSUPPORTED_OPERATION while every other command keeps working. Only portrait
has been checked on hardware, so other orientations are refused the same way.

A finger whose first waypoint is after 0 touches down late, which is how a
second finger joins a gesture already under way.
…iPhone

gesture-pinch, gesture-rotate and gesture-custom now run on a physical iPhone
through the runner's touchPaths command, and e2e pinch/rotate steps no longer
fail upfront on one.

- gesture-pinch and gesture-rotate build two timed finger paths at the same
  16 ms cadence the simulator and emulator paths use. The geometry moves to
  gesture-utils so Harmony, the simulator loops and the iPhone share it.
- gesture-custom keeps a plain Down/Up (press-hold, drag, long-press pickup)
  on XCUICoordinate, whose hold semantics a pickup needs, and plays every other
  Down/Move/Up train, a second finger or Move waypoints, as finger paths.
  Only a train that is not a Down, Moves, then an Up is refused.
- Drop the describe header's "no two-finger gestures" line and the e2e gate,
  and update the docs, rules and skills that repeated the limit.
… the display-aware record

- Check the orientation the app draws in, read from XCUIApplication's
  interfaceOrientation, instead of the way the phone is held. A portrait-only
  app on a phone turned sideways is no longer refused, and an app drawn in
  landscape still is, since only portrait has been checked on hardware. An
  orientation the runner cannot read is refused too, rather than guessed.
- Build the event record with initWithName:displayID:interfaceOrientation:
  when XCTest and the main screen offer a display id, and fall back to
  initWithName:interfaceOrientation: otherwise. Either one satisfies the
  availability check.
- Say "the app drawn in portrait" instead of "the phone held in portrait" in
  the tool descriptions, docs, skills and PROTOCOL.md.
…are gesture docs

- Type the runner mocks' touchPaths requests in the gesture-custom and
  touch-paths tests, so typecheck:tests passes again.
- run-sequence: list gesture-pinch and gesture-rotate among the steps that run
  on a physical iOS device.
- gesture-pinch and gesture-rotate: document reactivated: true and the refusal
  when the app is not drawn in portrait, like the other gesture tools. Drop the
  runner sentence from gesture-rotate so the two descriptions stay distinct
  enough for the description-quality gate.
- physical-ios-devices docs: the portrait rule also covers single-finger custom
  paths, an Xcode without the synthesizer reports the gestures as unavailable,
  taps, swipes, long presses and drags keep working, and a gesture uses at most
  two fingers.

@hubgan hubgan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR! Thanks for your contribution!

@hubgan
hubgan merged commit cd2f2c6 into software-mansion:main Oct 30, 2026
12 checks passed
@camchis

camchis commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @hubgan!

hubgan added a commit that referenced this pull request Nov 2, 2026
…physical iPhone (#1715)

Makes a `gesture-custom` path train on a physical iPhone hold the finger
through each event's `delayMs`, then move it, as a simulator does.

## Why

`delayMs` is the wait before an event, with the finger where it was. On
a simulator the finger rests, then moves to the event's point. On a
physical iPhone, path trains go through the runner's `touchPaths`, and
XCTest glides a finger between consecutive waypoints, so the finger
started moving as soon as the delay began.

A press, a 1 s wait, and a move:

```json
[
  { "type": "Down", "x": 0.5, "y": 0.47 },
  { "type": "Move", "x": 0.5, "y": 0.3, "delayMs": 1000 },
  { "type": "Up", "x": 0.5, "y": 0.3 }
]
```

On a link in Safari this opens the link menu on a simulator. On an
iPhone it scrolled the page, because the finger never held still. Before
#1664 the phone refused this train; since then it played a different
gesture without saying so, and an e2e step recorded on a simulator
replayed as a scroll on a phone.

## What changes

- `trainPaths` adds a waypoint at the finger's last point 1 ms before
each event that moves it. The finger rests through the delay, then
reaches the event's point. Both fingers get the same treatment.
- No rest is added when the finger stays put, or when the delay is 1 ms
or less.
- A train with few, far-apart events now moves in steps on an iPhone, as
it does on a simulator. `interpolate` still adds intermediate frames for
smooth motion.
- Unchanged: pinch and rotate (their paths come from `pinchPaths` and
`rotatePaths`), and the plain Down/Up and long-press pickup trains,
which still go through XCUICoordinate.

## Docs and tests

- No docs change: this is the documented `delayMs` behaviour ("Delay
before this event"), and the `gesture-custom` description already points
to `interpolate` for smooth motion.
- `ios-device-touch-paths.test.ts`: the rest before a move, and no rest
for a finger that stays put or a 1 ms delay. Updated waypoints for the
existing trains.
- `gesture-custom.test.ts`: updated waypoints for the two-finger,
waypoint and interpolate trains.
pFornagiel added a commit that referenced this pull request Nov 3, 2026
Conflicts were resolved by keeping the trimmed text from this branch and
adding the new facts from main in shorter form:

- Physical iPhone pinch/rotate (#1664): gesture-pinch, gesture-rotate and
  gesture-custom state the portrait limit and the reactivated flag;
  run-sequence lists pinch/rotate for physical iOS; the iOS-device, e2e and
  QA skills drop the "no two-finger gestures" lines.
- Remote simulator recording (#971): the screen-recording tools accept
  remote simulators and say that stop downloads the mp4; the skill lists
  remote simulators as recordable and notes that landscape plays sideways.
- rotate refuses Android TV (#1354): rotate becomes createRotateTool and
  says it fails on a TV target.
latekvo added a commit that referenced this pull request Nov 4, 2026
…at never go idle (#1722)

On a physical iPhone, a gesture on a screen that never goes idle
(Bluesky's profile: expo-blur keeps a paused `UIViewPropertyAnimator`)
took 60-120 s and then failed with `RUNNER_BUSY`. Two things caused it:

- **Slow:** XCTest waits up to 60 s before and after every event for the
app to go idle.
- **Repeated:** the MCP adapter aborts a call after 30 s and sends it
again. The tool-server keeps running each aborted call, so one
`gesture-tap` landed 2-5 times.

**Fix:**
- The runner skips XCTest's idle waits and instead waits at most 2 s for
the app to settle before each event.
- `gesture-tap`, `gesture-swipe`, `gesture-custom`, `gesture-pinch`,
`gesture-rotate` and `button` are `longRunning`, as `keyboard` already
is.

| Stalled screen, one MCP `gesture-tap` | main | this PR |
|---|---|---|
| time | 135 s, then `RUNNER_BUSY` | ~5 s, ok |
| physical taps | 2 | 1 |
| text entry (focus / type) | 75 s / 55 s, both timed out | 2.3 s / 3.4
s, text landed |

> [!NOTE]
> **This uses private XCTest API.** There's no screenshotting or
sleep-polling. `ArgentRunnerSession+Quiescence.swift`:
> 1. Calls the same check XCTest runs on its own (`applicationImpl` →
`currentProcess` → `_initiateQuiescenceChecksIncludingAnimationsIdle:`
and `_makeQuiescenceExpectation`), but waits on it for at most 2 s
instead of 60 s.
> 2. Sends the event inside `-[XCUIApplication
_performWithInteractionOptions:block:]` with options `0b11` (skip the
pre-event and post-event waits). This is the same SPI agent-device uses.
>
> Selectors are resolved with `NSSelectorFromString` and called through
`unsafeBitCast` IMPs, and each one is checked with `responds(to:)`
first. If an Xcode drops one, the gesture falls back to XCTest's normal
60 s waits rather than crashing.
>
> **Cost:** on a screen that never goes idle, every gesture now waits
the full 2 s. On main it waited 60-120 s. Skipping the wait entirely
(0.4 s) lost taps sent right after a navigation (0/5 landed).

Docs: no docs-site change. `longRunning` is internal, and the runner's
PROTOCOL.md/README are updated.

<details>
<summary>Repro and verification</summary>

No physical iPhone was used. The real runner ran on an iOS 18.5
simulator, with stub `xcrun`/`xcodebuild` and a fake usbmuxd making the
tool-server treat it as a phone. The tool-server, MCP adapter and runner
were the real code. The fixture app stalls XCTest with either a paused
`UIViewPropertyAnimator` on a visual-effect view or a repeating
`UIView.animate`.

Direct to the runner:

| Case | main | this PR |
|---|---|---|
| Stalled screen, 3 taps | 60.5 s ok; 75.1 s `COMMAND_TIMED_OUT`; then
`RUNNER_BUSY` | 0.41 s, then 2.32-2.35 s each; 6/6 landed (both stall
types) |
| Stalled screen, focus / type / delete / return | 75 / 55 / 30 / 30 s,
all timed out | 2.34 / 3.36 / 3.32 / 2.42 s, all ok |
| Stalled screen, doubleTap / longPress / drag | - | 2.48 / 3.10 / 2.80
s, all landed |
| Normal screen: open modal, wait 100 ms, close (5 runs) | 5/5, close
took 1.0-1.1 s | 5/5, close took 0.75-0.82 s |
| `touchPaths` pinch (from #1664), stalled vs normal | - | ~0.74 s on
both, 3/3 registered (synthesized directly, never waited for idle) |

Through MCP, first tap of a session with the runner build padded to 45
s: main gave 2 tool calls and 2 taps (the app saw a double tap); this PR
gives 1 call and 1 tap.

Local checks: `npm run build`, `typecheck:tests`, prettier and eslint
pass. vitest passed 826 tests across the gesture, button, keyboard,
http, ios-device, ios-physical and run-sequence suites. The runner
builds for a real device (`build-for-testing generic/platform=iOS`).
`ios-device-gestures-long-running.test.ts` fails when any single
`longRunning` flag is removed.

Not fixed here:
- A first runner build longer than 300 s is still replayed (#1632).
- Bringing a backgrounded app to the foreground still uses XCTest's
normal waits, capped at about 15 s.
- #1528's `element.tap()` would need the same `skippingQuiescence`
wrapper.
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants