Repository navigation
feat(ios-device): pinch, rotate and multi-finger paths on a physical iPhone - #1664
Merged
hubgan merged 4 commits intoOct 30, 2026
Merged
Conversation
…stures XCUICoordinate draws one straight press-then-drag, so the runner could not play a pinch, a rotation or a path with waypoints. Add `touchPaths`: one timed waypoint path per finger (1 to 5), played at once as a single event record through XCTest's own XCPointerEventPath / XCSynthesizedEventRecord. Those classes are private, so ArgentTouchSynthesizer resolves them by name at runtime and checks every selector first. Nothing links against a private symbol, and an Xcode that drops one makes the command answer UNSUPPORTED_OPERATION while every other command keeps working. Only portrait has been checked on hardware, so other orientations are refused the same way. A finger whose first waypoint is after 0 touches down late, which is how a second finger joins a gesture already under way.
…iPhone gesture-pinch, gesture-rotate and gesture-custom now run on a physical iPhone through the runner's touchPaths command, and e2e pinch/rotate steps no longer fail upfront on one. - gesture-pinch and gesture-rotate build two timed finger paths at the same 16 ms cadence the simulator and emulator paths use. The geometry moves to gesture-utils so Harmony, the simulator loops and the iPhone share it. - gesture-custom keeps a plain Down/Up (press-hold, drag, long-press pickup) on XCUICoordinate, whose hold semantics a pickup needs, and plays every other Down/Move/Up train, a second finger or Move waypoints, as finger paths. Only a train that is not a Down, Moves, then an Up is refused. - Drop the describe header's "no two-finger gestures" line and the e2e gate, and update the docs, rules and skills that repeated the limit.
… the display-aware record - Check the orientation the app draws in, read from XCUIApplication's interfaceOrientation, instead of the way the phone is held. A portrait-only app on a phone turned sideways is no longer refused, and an app drawn in landscape still is, since only portrait has been checked on hardware. An orientation the runner cannot read is refused too, rather than guessed. - Build the event record with initWithName:displayID:interfaceOrientation: when XCTest and the main screen offer a display id, and fall back to initWithName:interfaceOrientation: otherwise. Either one satisfies the availability check. - Say "the app drawn in portrait" instead of "the phone held in portrait" in the tool descriptions, docs, skills and PROTOCOL.md.
…are gesture docs - Type the runner mocks' touchPaths requests in the gesture-custom and touch-paths tests, so typecheck:tests passes again. - run-sequence: list gesture-pinch and gesture-rotate among the steps that run on a physical iOS device. - gesture-pinch and gesture-rotate: document reactivated: true and the refusal when the app is not drawn in portrait, like the other gesture tools. Drop the runner sentence from gesture-rotate so the two descriptions stay distinct enough for the description-quality gate. - physical-ios-devices docs: the portrait rule also covers single-finger custom paths, an Xcode without the synthesizer reports the gestures as unavailable, taps, swipes, long presses and drags keep working, and a gesture uses at most two fingers.
hubgan
approved these changes
Oct 30, 2026
hubgan
left a comment
Collaborator
There was a problem hiding this comment.
Great PR! Thanks for your contribution!
Contributor
Author
|
Thanks @hubgan! |
This was referenced Oct 30, 2026
hubgan
added a commit
that referenced
this pull request
Nov 2, 2026
…physical iPhone (#1715) Makes a `gesture-custom` path train on a physical iPhone hold the finger through each event's `delayMs`, then move it, as a simulator does. ## Why `delayMs` is the wait before an event, with the finger where it was. On a simulator the finger rests, then moves to the event's point. On a physical iPhone, path trains go through the runner's `touchPaths`, and XCTest glides a finger between consecutive waypoints, so the finger started moving as soon as the delay began. A press, a 1 s wait, and a move: ```json [ { "type": "Down", "x": 0.5, "y": 0.47 }, { "type": "Move", "x": 0.5, "y": 0.3, "delayMs": 1000 }, { "type": "Up", "x": 0.5, "y": 0.3 } ] ``` On a link in Safari this opens the link menu on a simulator. On an iPhone it scrolled the page, because the finger never held still. Before #1664 the phone refused this train; since then it played a different gesture without saying so, and an e2e step recorded on a simulator replayed as a scroll on a phone. ## What changes - `trainPaths` adds a waypoint at the finger's last point 1 ms before each event that moves it. The finger rests through the delay, then reaches the event's point. Both fingers get the same treatment. - No rest is added when the finger stays put, or when the delay is 1 ms or less. - A train with few, far-apart events now moves in steps on an iPhone, as it does on a simulator. `interpolate` still adds intermediate frames for smooth motion. - Unchanged: pinch and rotate (their paths come from `pinchPaths` and `rotatePaths`), and the plain Down/Up and long-press pickup trains, which still go through XCUICoordinate. ## Docs and tests - No docs change: this is the documented `delayMs` behaviour ("Delay before this event"), and the `gesture-custom` description already points to `interpolate` for smooth motion. - `ios-device-touch-paths.test.ts`: the rest before a move, and no rest for a finger that stays put or a 1 ms delay. Updated waypoints for the existing trains. - `gesture-custom.test.ts`: updated waypoints for the two-finger, waypoint and interpolate trains.
pFornagiel
added a commit
that referenced
this pull request
Nov 3, 2026
Conflicts were resolved by keeping the trimmed text from this branch and adding the new facts from main in shorter form: - Physical iPhone pinch/rotate (#1664): gesture-pinch, gesture-rotate and gesture-custom state the portrait limit and the reactivated flag; run-sequence lists pinch/rotate for physical iOS; the iOS-device, e2e and QA skills drop the "no two-finger gestures" lines. - Remote simulator recording (#971): the screen-recording tools accept remote simulators and say that stop downloads the mp4; the skill lists remote simulators as recordable and notes that landscape plays sideways. - rotate refuses Android TV (#1354): rotate becomes createRotateTool and says it fails on a TV target.
latekvo
added a commit
that referenced
this pull request
Nov 4, 2026
…at never go idle (#1722) On a physical iPhone, a gesture on a screen that never goes idle (Bluesky's profile: expo-blur keeps a paused `UIViewPropertyAnimator`) took 60-120 s and then failed with `RUNNER_BUSY`. Two things caused it: - **Slow:** XCTest waits up to 60 s before and after every event for the app to go idle. - **Repeated:** the MCP adapter aborts a call after 30 s and sends it again. The tool-server keeps running each aborted call, so one `gesture-tap` landed 2-5 times. **Fix:** - The runner skips XCTest's idle waits and instead waits at most 2 s for the app to settle before each event. - `gesture-tap`, `gesture-swipe`, `gesture-custom`, `gesture-pinch`, `gesture-rotate` and `button` are `longRunning`, as `keyboard` already is. | Stalled screen, one MCP `gesture-tap` | main | this PR | |---|---|---| | time | 135 s, then `RUNNER_BUSY` | ~5 s, ok | | physical taps | 2 | 1 | | text entry (focus / type) | 75 s / 55 s, both timed out | 2.3 s / 3.4 s, text landed | > [!NOTE] > **This uses private XCTest API.** There's no screenshotting or sleep-polling. `ArgentRunnerSession+Quiescence.swift`: > 1. Calls the same check XCTest runs on its own (`applicationImpl` → `currentProcess` → `_initiateQuiescenceChecksIncludingAnimationsIdle:` and `_makeQuiescenceExpectation`), but waits on it for at most 2 s instead of 60 s. > 2. Sends the event inside `-[XCUIApplication _performWithInteractionOptions:block:]` with options `0b11` (skip the pre-event and post-event waits). This is the same SPI agent-device uses. > > Selectors are resolved with `NSSelectorFromString` and called through `unsafeBitCast` IMPs, and each one is checked with `responds(to:)` first. If an Xcode drops one, the gesture falls back to XCTest's normal 60 s waits rather than crashing. > > **Cost:** on a screen that never goes idle, every gesture now waits the full 2 s. On main it waited 60-120 s. Skipping the wait entirely (0.4 s) lost taps sent right after a navigation (0/5 landed). Docs: no docs-site change. `longRunning` is internal, and the runner's PROTOCOL.md/README are updated. <details> <summary>Repro and verification</summary> No physical iPhone was used. The real runner ran on an iOS 18.5 simulator, with stub `xcrun`/`xcodebuild` and a fake usbmuxd making the tool-server treat it as a phone. The tool-server, MCP adapter and runner were the real code. The fixture app stalls XCTest with either a paused `UIViewPropertyAnimator` on a visual-effect view or a repeating `UIView.animate`. Direct to the runner: | Case | main | this PR | |---|---|---| | Stalled screen, 3 taps | 60.5 s ok; 75.1 s `COMMAND_TIMED_OUT`; then `RUNNER_BUSY` | 0.41 s, then 2.32-2.35 s each; 6/6 landed (both stall types) | | Stalled screen, focus / type / delete / return | 75 / 55 / 30 / 30 s, all timed out | 2.34 / 3.36 / 3.32 / 2.42 s, all ok | | Stalled screen, doubleTap / longPress / drag | - | 2.48 / 3.10 / 2.80 s, all landed | | Normal screen: open modal, wait 100 ms, close (5 runs) | 5/5, close took 1.0-1.1 s | 5/5, close took 0.75-0.82 s | | `touchPaths` pinch (from #1664), stalled vs normal | - | ~0.74 s on both, 3/3 registered (synthesized directly, never waited for idle) | Through MCP, first tap of a session with the runner build padded to 45 s: main gave 2 tool calls and 2 taps (the app saw a double tap); this PR gives 1 call and 1 tap. Local checks: `npm run build`, `typecheck:tests`, prettier and eslint pass. vitest passed 826 tests across the gesture, button, keyboard, http, ios-device, ios-physical and run-sequence suites. The runner builds for a real device (`build-for-testing generic/platform=iOS`). `ios-device-gestures-long-running.test.ts` fails when any single `longRunning` flag is removed. Not fixed here: - A first runner build longer than 300 s is still replayed (#1632). - Bringing a backgrounded app to the foreground still uses XCTest's normal waits, capped at about 15 s. - #1528's `element.tap()` would need the same `skippingQuiescence` wrapper. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
touchPathscommand to the iOS device runner: one timed waypoint path per finger (1 to 5), played at once as a single event record through XCTest's event synthesizer.gesture-pinch,gesture-rotateandgesture-customnow run on a physical iPhone through it. e2epinch/rotatesteps no longer fail upfront there.gesture-custom(press-hold, drag, long-press pickup) stays onXCUICoordinate, whose hold semantics a pickup needs. Only a second finger orMovewaypoints take the new path.describe's header.Why
On hardware the runner could only draw one straight press-then-drag, so an agent testing an app with a pinch-to-zoom view, a map, a canvas, or any custom multi-touch handler had to give up or drive the app's zoom buttons. The docs and tool descriptions say "XCTest exposes no two-finger coordinate API". That is true of the public
XCUICoordinateAPI, but XCTest builds its own touches fromXCPointerEventPathandXCSynthesizedEventRecord, and WebDriverAgent has used them for W3C multi-finger actions for years.How it works
ArgentTouchSynthesizer(Objective-C) looks both classes up withNSClassFromStringand checks every selector withinstancesRespondToSelector:before use. Nothing links against a private symbol.touchPathsanswersUNSUPPORTED_OPERATIONwith the name of what is missing. Every other command keeps working, and single-finger gestures are unaffected.fingers: [{ points: [{ x, y, tMs }] }], absolute points in the app's space liketapanddrag. A finger whose first waypoint is later than 0 touches down late, which is how a second finger joins a gesture in progress. Documented inPROTOCOL.md.gesture-utilsso Harmony, the simulator loops and the iPhone path share one implementation. Frames are 16 ms apart, the cadence the simulator and emulator paths already use.initWithName:displayID:interfaceOrientation:when XCTest and the main screen offer a display id, and with the olderinitWithName:interfaceOrientation:otherwise. Either one passes the availability check, so an Xcode that keeps only one of them still works.XCUIApplication'sinterfaceOrientation), not the way the phone is held, so a portrait-only app on a phone turned sideways is not refused. An app drawn in landscape is refused because I have not checked landscape on hardware, and a touch that lands somewhere else without an error is worse than a refusal. An orientation the runner cannot read is refused too, rather than guessed. Lifting this needs a point mapping into the panel's portrait space: feat(apple): admit two-finger gestures on physical iOS devices callstack/agent-device#3838 has one, and I can add it once someone has verified landscape on a phone.Verified on hardware
iPhone 12, iOS 27.0.1, Xcode 27.1, driven through the tool-server built from this branch with
ARGENT_IOS_RUNNER_PROJECTpointing at it, Safari on a Wikipedia article. There were two runs, because the third commit changed which record initializer is used.First two commits (
initWithName:interfaceOrientation:):gesture-pinch0.1 to 0.6 over 400 msgesture-pinch0.6 to 0.1gesture-rotaterotated: true(Safari has no rotate handler, so there is nothing visible)gesture-custom, one finger, 3Movewaypointsgesture-custom, second finger joining lategesture-custom, plain Down/Updragpathgesture-custom, train starting withMoveThird commit (
initWithName:displayID:interfaceOrientation:, display id 1, orientation read as portrait). A temporary log line, removed before committing, confirmed both values. I then re-ran on the committed code:gesture-custom, four-event train at the off-centre "Bell Labs" link (takestouchPaths, notdrag)gesture-pinchout, then ingesture-rotaterotated: truegesture-custom, train starting withMoveI confirmed the synthesizer in a standalone XCUITest before wiring it into the runner.
Not verified: other Xcode or iOS versions, iPads (not supported by the runner), landscape (refused), a portrait-only app on a phone held sideways (accepted by the check but not tried), any app other than Safari, and timing fidelity. In the standalone probe a 0.6 s script took 0.96 s end to end, so expect gestures to run somewhat longer than their nominal duration.
Tests
ios-device-touch-paths.test.ts(new): exact waypoints for pinch (spread, odd duration, angle plus drifting centre, off-screen clamp), rotate, and trains (delay spacing, late and early second finger, refusals), plus thetouchPathswire shape and the pinch/rotate tools on a device.gesture-custom.test.ts: the "rejects what XCTest cannot replay" cases are replaced with the new behaviour and the remaining refusals.flow-pinch/flow-rotate/describe-format-tree/ capability ratchet updated for the new behaviour.npm run build,npm run knipand eslint on the changed files pass. The full tool-server suite has 3 failures that are identical on an untouchedmainin my environment (two real-ffmpeg letterbox tests and thee2e-tool-aliaseslisting test), so I left them alone.Related
ArgentRunnerSession+Gestures.swiftandRunnerProtocol.swift(fix: add explicit native taps for physical iOS #1528 alsoPROTOCOL.mdandrunner-commands.ts). The features are independent, but expect small textual conflicts in whichever merges second. fix(ios): guard foldable capture and transposed window coordinates #1624 changes how window coordinates are mapped on foldables, so it is worth a look alongside this PR's portrait-only rule.simulator-server.