Skip to content

Commit e35d8de

Browse files
camchisCameron Chisholm
andauthored
feat(gesture): add gesture tap with 2 to 5 simultaneous contacts (#3847)
* refactor(gesture): rename the two-contact plan topology to multi Multi-contact guards now test topology !== 'single', so a later multi-contact intent reaches the same checks. No behaviour change. * feat(gesture): add gesture tap with 2 to 5 simultaneous contacts gesture tap touches 2 to 5 distinct points at once and lifts them together after --hold-ms (default 50 ms). --count and --pause-ms repeat the same contacts: each repetition plans against a fresh viewport, the series is budgeted against the request envelope, and an aborted request stops between repetitions. The CLI, .ad replay, Node.js client (tapGesture) and MCP accept it; gesture now refuses flags a kind does not read. Taps admit through the multi-touch tier; WebDriver maps them to its multi-pointer capability. * feat(apple): accept multi-finger tap plans in the iOS runner The runner admits a multi-topology tap with 2 to 5 stationary contacts on distinct points; pan, pinch, rotate and transform still need exactly two. The private XCTest bridge already synthesizes one path per contact, so it is unchanged. * feat(android): inject multi-finger tap plans from the snapshot helper The helper accepts a tap plan with 2 to 5 stationary contacts and generalizes its pointer schedule to N contacts: extra contacts go down in ascending index before the first sample and lift in descending order after the last, so the changing contact is always the highest active index. Transition spacing shrinks below 8 ms only when needed to fit every contact before the first sample; two-contact schedules are unchanged. Sample offsets are now checked for every path, not just the first two. A contracts test pins the 5-contact cap across TypeScript, Swift and Java. * docs(gesture): document gesture tap and amend ADR 0013 GestureLab reports multi-tap rounds and the most contacts held in the last round, so a tap has an app-observed postcondition. * fix(gesture): propagate cancellation without recording partial tap series * refactor(gesture): keep cancellation checks at daemon boundaries --------- Co-authored-by: Cameron Chisholm <[email protected]>
1 parent 421dd7e commit e35d8de

74 files changed

Lines changed: 2022 additions & 149 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎android/snapshot-helper/README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,14 +77,16 @@ adb shell am instrument -w -e mode gesture -e payloadBase64 "$PAYLOAD" \
7777

7878
The `gesture` payload is a base64-encoded JSON object using protocol `android-touch-plan-v1`:
7979

80-
- `kind`: `swipe` (one pointer) or `transform` (two pointers, e.g. pinch/rotate)
80+
- `kind`: `swipe` (one pointer), `transform` (two pointers, e.g. pinch/rotate), or `tap` (2 to 5
81+
stationary pointers on distinct points)
8182
- `durationMs`: integer, `0`-`120000`
8283
- `pointers`: array of `{ pointerId, samples }`, `pointerId`s ordered from `0`, one pointer for
83-
`swipe` and exactly two for `transform`
84+
`swipe`, exactly two for `transform`, and 2 to 5 for `tap`
8485
- each pointer's `samples` is `{ offsetMs, x, y }[]` with at least two entries; `offsetMs` values
8586
must be strictly increasing (equal offsets are only allowed when `durationMs` is `0`), the first
86-
sample's `offsetMs` must be `0`, and the last must equal `durationMs`. For `transform` gestures,
87-
both pointers must share the same `offsetMs` sequence.
87+
sample's `offsetMs` must be `0`, and the last must equal `durationMs`. For `transform` and `tap`
88+
gestures, every pointer must share the same `offsetMs` sequence, and a `tap` pointer must not
89+
move.
8890

8991
## Persistent Session
9092

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/PointerEventSchedule.java‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,14 @@
33
import java.util.ArrayList;
44
import java.util.List;
55

6-
/** Defines the platform-independent event order consumed by Android touch injection. */
6+
/**
7+
* Defines the platform-independent event order consumed by Android touch injection.
8+
*
9+
* <p>Extra contacts go down in ascending index order before the first planned move sample and lift
10+
* in descending order 8 ms apart after the last one, so the active contacts are always indices 0 to
11+
* {@code pointerCount - 1} and the changing contact is always the highest of them. With two
12+
* contacts this is the original single-transition schedule.
13+
*/
714
final class PointerEventSchedule {
815
private static final long POINTER_TRANSITION_DELAY_MS = 8;
916

@@ -39,20 +46,21 @@ static final class Step {
3946
private PointerEventSchedule() {}
4047

4148
static List<Step> create(int pointerCount, long[] sampleOffsetsMs) {
42-
if (pointerCount != 1 && pointerCount != 2) {
43-
throw new IllegalArgumentException("Pointer event schedule requires one or two pointers");
49+
if (pointerCount < 1 || pointerCount > TouchPlan.MAX_CONTACTS) {
50+
throw new IllegalArgumentException(
51+
"Pointer event schedule requires 1 to " + TouchPlan.MAX_CONTACTS + " pointers");
4452
}
4553
if (sampleOffsetsMs.length < 2) {
4654
throw new IllegalArgumentException("Pointer event schedule requires at least two samples");
4755
}
4856

4957
int lastIndex = sampleOffsetsMs.length - 1;
50-
List<Step> steps = new ArrayList<>(sampleOffsetsMs.length + (pointerCount == 1 ? 1 : 3));
58+
long lastOffset = sampleOffsetsMs[lastIndex];
59+
long transitionMs = transitionSpacing(pointerCount, sampleOffsetsMs[1]);
60+
List<Step> steps = new ArrayList<>(sampleOffsetsMs.length + 2 * pointerCount);
5161
steps.add(new Step(Action.DOWN, 0, 1, 0, true));
52-
if (pointerCount == 2) {
53-
long pointerDownOffset =
54-
Math.max(1, Math.min(POINTER_TRANSITION_DELAY_MS, sampleOffsetsMs[1] - 1));
55-
steps.add(new Step(Action.POINTER_DOWN, 0, 2, pointerDownOffset, true));
62+
for (int contact = 2; contact <= pointerCount; contact += 1) {
63+
steps.add(new Step(Action.POINTER_DOWN, 0, contact, (contact - 1) * transitionMs, true));
5664
}
5765
for (int sampleIndex = 1; sampleIndex <= lastIndex; sampleIndex += 1) {
5866
steps.add(
@@ -63,13 +71,19 @@ static List<Step> create(int pointerCount, long[] sampleOffsetsMs) {
6371
sampleOffsetsMs[sampleIndex],
6472
pointerCount == 1 || sampleIndex == lastIndex));
6573
}
66-
if (pointerCount == 2) {
67-
steps.add(
68-
new Step(Action.POINTER_UP, lastIndex, 2, sampleOffsetsMs[lastIndex], true));
74+
for (int contact = pointerCount; contact >= 2; contact -= 1) {
75+
long offset = lastOffset + (pointerCount - contact) * POINTER_TRANSITION_DELAY_MS;
76+
steps.add(new Step(Action.POINTER_UP, lastIndex, contact, offset, true));
6977
}
70-
long finalUpOffset =
71-
sampleOffsetsMs[lastIndex] + (pointerCount == 2 ? POINTER_TRANSITION_DELAY_MS : 0);
78+
long finalUpOffset = lastOffset + (pointerCount - 1) * POINTER_TRANSITION_DELAY_MS;
7279
steps.add(new Step(Action.UP, lastIndex, 1, finalUpOffset, true));
7380
return steps;
7481
}
82+
83+
/** The gap between touch-downs: up to 8 ms, shrunk so every contact is down before the first move. */
84+
private static long transitionSpacing(int pointerCount, long firstSampleOffsetMs) {
85+
if (pointerCount == 1) return 0;
86+
long fitting = (firstSampleOffsetMs - 1) / (pointerCount - 1);
87+
return Math.max(1, Math.min(POINTER_TRANSITION_DELAY_MS, fitting));
88+
}
7589
}

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/TouchPlan.java‎

Lines changed: 45 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,11 @@
55
import org.json.JSONArray;
66
import org.json.JSONObject;
77

8-
/** Validated planned-touch payload: one or two complete pointer trajectories. */
8+
/** Validated planned-touch payload: one, two, or (for a tap) up to five pointer trajectories. */
99
final class TouchPlan {
1010
static final String PAYLOAD_PROTOCOL = "android-touch-plan-v1";
11+
/** Mirrors GESTURE_MAX_CONTACTS in packages/contracts/src/gesture-plan-types.ts. */
12+
static final int MAX_CONTACTS = 5;
1113
private static final int MIN_DURATION_MS = 0;
1214
private static final int MAX_DURATION_MS = 120_000;
1315

@@ -33,7 +35,7 @@ static TouchPlan parseBase64(String payloadBase64) throws Exception {
3335
throw new IllegalArgumentException("Unsupported protocol: " + protocol);
3436
}
3537
String kind = payload.getString("kind");
36-
if (!"swipe".equals(kind) && !"transform".equals(kind)) {
38+
if (!"swipe".equals(kind) && !"transform".equals(kind) && !"tap".equals(kind)) {
3739
throw new IllegalArgumentException("Unsupported kind: " + kind);
3840
}
3941
int durationMs = requireDuration(payload.getInt("durationMs"));
@@ -43,11 +45,7 @@ static TouchPlan parseBase64(String payloadBase64) throws Exception {
4345

4446
private static PointerPath[] readPointers(JSONArray pointers, String kind, int durationMs)
4547
throws Exception {
46-
int expectedCount = "swipe".equals(kind) ? 1 : 2;
47-
if (pointers.length() != expectedCount) {
48-
throw new IllegalArgumentException(
49-
"Planned " + kind + " gesture requires exactly " + expectedCount + " pointer paths");
50-
}
48+
int expectedCount = expectedPointerCount(kind, pointers.length());
5149
PointerPath[] result = new PointerPath[expectedCount];
5250
for (int pointerIndex = 0; pointerIndex < expectedCount; pointerIndex += 1) {
5351
JSONObject pointer = pointers.getJSONObject(pointerIndex);
@@ -84,21 +82,57 @@ private static PointerPath[] readPointers(JSONArray pointers, String kind, int d
8482
}
8583
result[pointerIndex] = new PointerPath(parsed);
8684
}
87-
if (expectedCount == 2) assertMatchingSamples(result[0], result[1]);
85+
for (int pointerIndex = 1; pointerIndex < expectedCount; pointerIndex += 1) {
86+
assertMatchingSamples(result[0], result[pointerIndex]);
87+
}
88+
if ("tap".equals(kind)) assertStationaryDistinctContacts(result);
8889
return result;
8990
}
9091

91-
private static void assertMatchingSamples(PointerPath first, PointerPath second) {
92-
if (first.samples.length != second.samples.length) {
92+
private static int expectedPointerCount(String kind, int actual) {
93+
if ("tap".equals(kind)) {
94+
if (actual < 2 || actual > MAX_CONTACTS) {
95+
throw new IllegalArgumentException(
96+
"Planned tap gesture requires 2 to " + MAX_CONTACTS + " pointer paths");
97+
}
98+
return actual;
99+
}
100+
int expected = "swipe".equals(kind) ? 1 : 2;
101+
if (actual != expected) {
102+
throw new IllegalArgumentException(
103+
"Planned " + kind + " gesture requires exactly " + expected + " pointer paths");
104+
}
105+
return expected;
106+
}
107+
108+
private static void assertMatchingSamples(PointerPath first, PointerPath other) {
109+
if (first.samples.length != other.samples.length) {
93110
throw new IllegalArgumentException("Planned pointer paths must have matching samples");
94111
}
95112
for (int index = 0; index < first.samples.length; index += 1) {
96-
if (first.samples[index].offsetMs != second.samples[index].offsetMs) {
113+
if (first.samples[index].offsetMs != other.samples[index].offsetMs) {
97114
throw new IllegalArgumentException("Planned pointer sample offsets must match");
98115
}
99116
}
100117
}
101118

119+
private static void assertStationaryDistinctContacts(PointerPath[] pointers) {
120+
for (int pointerIndex = 0; pointerIndex < pointers.length; pointerIndex += 1) {
121+
PointerSample start = pointers[pointerIndex].samples[0];
122+
for (PointerSample sample : pointers[pointerIndex].samples) {
123+
if (sample.x != start.x || sample.y != start.y) {
124+
throw new IllegalArgumentException("Planned tap contacts must not move");
125+
}
126+
}
127+
for (int earlier = 0; earlier < pointerIndex; earlier += 1) {
128+
PointerSample other = pointers[earlier].samples[0];
129+
if (other.x == start.x && other.y == start.y) {
130+
throw new IllegalArgumentException("Planned tap contacts must start on distinct points");
131+
}
132+
}
133+
}
134+
}
135+
102136
private static int requireDuration(int value) {
103137
if (value < MIN_DURATION_MS || value > MAX_DURATION_MS) {
104138
throw new IllegalArgumentException("durationMs must be between 0 and 120000");

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/TouchPlanInjector.java‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,17 +12,17 @@ private TouchPlanInjector() {}
1212
static int inject(UiAutomation automation, TouchPlan plan) {
1313
long downTime = SystemClock.uptimeMillis();
1414
long eventTime = downTime;
15-
PointerState active = pointerStateAt(plan, 0).firstOnly();
15+
PointerState active = pointerStateAt(plan, 0).firstN(1);
1616
int count = 0;
1717
try {
1818
for (PointerEventSchedule.Step step :
1919
PointerEventSchedule.create(plan.pointers.length, sampleOffsets(plan))) {
20-
active = pointerStateAt(plan, step.sampleIndex);
21-
if (step.pointerCount == 1) active = active.firstOnly();
20+
active = pointerStateAt(plan, step.sampleIndex).firstN(step.pointerCount);
2221
eventTime = downTime + step.offsetMs;
2322
injectEvent(
2423
automation,
25-
motionEvent(downTime, eventTime, motionEventAction(step.action), active),
24+
motionEvent(
25+
downTime, eventTime, motionEventAction(step.action, step.pointerCount), active),
2626
step.waitForDispatch);
2727
count += 1;
2828
}
@@ -42,16 +42,18 @@ private static long[] sampleOffsets(TouchPlan plan) {
4242
return offsets;
4343
}
4444

45-
private static int motionEventAction(PointerEventSchedule.Action action) {
45+
/** The changing contact of a pointer transition is the highest active index. */
46+
private static int motionEventAction(PointerEventSchedule.Action action, int pointerCount) {
47+
int changingIndex = (pointerCount - 1) << MotionEvent.ACTION_POINTER_INDEX_SHIFT;
4648
switch (action) {
4749
case DOWN:
4850
return MotionEvent.ACTION_DOWN;
4951
case POINTER_DOWN:
50-
return MotionEvent.ACTION_POINTER_DOWN | (1 << MotionEvent.ACTION_POINTER_INDEX_SHIFT);
52+
return MotionEvent.ACTION_POINTER_DOWN | changingIndex;
5153
case MOVE:
5254
return MotionEvent.ACTION_MOVE;
5355
case POINTER_UP:
54-
return MotionEvent.ACTION_POINTER_UP | (1 << MotionEvent.ACTION_POINTER_INDEX_SHIFT);
56+
return MotionEvent.ACTION_POINTER_UP | changingIndex;
5557
case UP:
5658
return MotionEvent.ACTION_UP;
5759
default:
@@ -146,8 +148,10 @@ private static final class PointerState {
146148
this.y = y;
147149
}
148150

149-
PointerState firstOnly() {
150-
return new PointerState(new float[] {x[0]}, new float[] {y[0]});
151+
PointerState firstN(int pointerCount) {
152+
if (pointerCount == count) return this;
153+
return new PointerState(
154+
java.util.Arrays.copyOf(x, pointerCount), java.util.Arrays.copyOf(y, pointerCount));
151155
}
152156
}
153157
}

‎android/snapshot-helper/src/test/java/com/callstack/agentdevice/snapshothelper/PointerEventScheduleTest.java‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,53 @@ static void run() {
2525
"MOVE:2:2:32:true",
2626
"POINTER_UP:2:2:32:true",
2727
"UP:2:1:40:true");
28+
// A short two-contact plan keeps its original schedule: the second contact goes down before
29+
// the first move, and the last contact lifts 8 ms after the second.
30+
assertSteps(
31+
PointerEventSchedule.create(2, new long[] {0, 5, 11, 16}),
32+
"DOWN:0:1:0:true",
33+
"POINTER_DOWN:0:2:4:true",
34+
"MOVE:1:2:5:false",
35+
"MOVE:2:2:11:false",
36+
"MOVE:3:2:16:true",
37+
"POINTER_UP:3:2:16:true",
38+
"UP:3:1:24:true");
39+
// A five-contact 16 ms tap: every contact is down before the hold ends, 3 ms apart, and the
40+
// contacts lift in reverse order 8 ms apart, so the changing contact is the highest index.
41+
assertSteps(
42+
PointerEventSchedule.create(5, new long[] {0, 16}),
43+
"DOWN:0:1:0:true",
44+
"POINTER_DOWN:0:2:3:true",
45+
"POINTER_DOWN:0:3:6:true",
46+
"POINTER_DOWN:0:4:9:true",
47+
"POINTER_DOWN:0:5:12:true",
48+
"MOVE:1:5:16:true",
49+
"POINTER_UP:1:5:16:true",
50+
"POINTER_UP:1:4:24:true",
51+
"POINTER_UP:1:3:32:true",
52+
"POINTER_UP:1:2:40:true",
53+
"UP:1:1:48:true");
54+
// A three-contact tap with the default 50 ms hold keeps the full 8 ms spacing.
55+
assertSteps(
56+
PointerEventSchedule.create(3, new long[] {0, 50}),
57+
"DOWN:0:1:0:true",
58+
"POINTER_DOWN:0:2:8:true",
59+
"POINTER_DOWN:0:3:16:true",
60+
"MOVE:1:3:50:true",
61+
"POINTER_UP:1:3:50:true",
62+
"POINTER_UP:1:2:58:true",
63+
"UP:1:1:66:true");
64+
assertRejected(6);
65+
assertRejected(0);
66+
}
67+
68+
private static void assertRejected(int pointerCount) {
69+
try {
70+
PointerEventSchedule.create(pointerCount, new long[] {0, 16});
71+
} catch (IllegalArgumentException expected) {
72+
return;
73+
}
74+
throw new AssertionError("Expected " + pointerCount + " pointers to be rejected");
2875
}
2976

3077
private static void assertSteps(List<PointerEventSchedule.Step> actual, String... expected) {

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -341,6 +341,10 @@ struct RunnerGesturePoint: Codable {
341341
let y: Double
342342
}
343343

344+
/// The most simultaneous contacts a gesture plan carries. Mirrors `GESTURE_MAX_CONTACTS` in
345+
/// packages/contracts/src/gesture-plan-types.ts; a contracts test pins the two together.
346+
let runnerGestureMaxContacts = 5
347+
344348
/// One allowlisted coordinate gesture step inside a fused `sequence` command.
345349
/// `kind` is decoded as a raw String (not an enum) so the runner can return a clear
346350
/// INVALID_ARGS for an unknown kind instead of a generic decode failure.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift‎

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -461,12 +461,13 @@ extension RunnerTests {
461461
}
462462

463463
func plannedGestureValidationError(_ plan: RunnerGesturePlan) -> String? {
464-
guard plan.topology == "single" || plan.topology == "two" else {
465-
return "planned gesture topology must be single or two"
464+
guard plan.topology == "single" || plan.topology == "multi" else {
465+
return "planned gesture topology must be single or multi"
466466
}
467+
let isTap = plan.topology == "multi" && plan.intent == "tap"
467468
let supportedIntent = plan.topology == "single"
468469
? plan.intent == "fling" || plan.intent == "pan"
469-
: plan.intent == "pan" || plan.intent == "pinch" || plan.intent == "rotate"
470+
: isTap || plan.intent == "pan" || plan.intent == "pinch" || plan.intent == "rotate"
470471
|| plan.intent == "transform"
471472
guard supportedIntent else { return "planned gesture has unsupported intent for its topology" }
472473
if plan.topology == "single" {
@@ -485,8 +486,12 @@ extension RunnerTests {
485486
else {
486487
return "planned gesture viewport must be finite and positive"
487488
}
488-
let expectedPointerCount = plan.topology == "single" ? 1 : 2
489-
guard plan.pointers.count == expectedPointerCount else {
489+
let pointerCountMatches = plan.topology == "single"
490+
? plan.pointers.count == 1
491+
: isTap
492+
? (2...runnerGestureMaxContacts).contains(plan.pointers.count)
493+
: plan.pointers.count == 2
494+
guard pointerCountMatches else {
490495
return "planned gesture pointer count does not match topology"
491496
}
492497
for (index, pointer) in plan.pointers.enumerated() where pointer.pointerId != index {
@@ -523,7 +528,10 @@ extension RunnerTests {
523528
return "planned pointer paths must start at 0 and end at durationMs"
524529
}
525530
}
526-
if plan.topology == "two" {
531+
if isTap {
532+
return plannedTapValidationError(plan)
533+
}
534+
if plan.topology == "multi" {
527535
guard let firstStart = firstSamples.first?.point,
528536
let secondStart = plan.pointers[1].samples.first?.point,
529537
hypot(firstStart.x - secondStart.x, firstStart.y - secondStart.y) > 0
@@ -534,6 +542,24 @@ extension RunnerTests {
534542
return nil
535543
}
536544

545+
/// Tap contacts stay where they land, and two contacts on one point would merge into one touch.
546+
private func plannedTapValidationError(_ plan: RunnerGesturePlan) -> String? {
547+
var starts: [(x: Double, y: Double)] = []
548+
for pointer in plan.pointers {
549+
guard let start = pointer.samples.first?.point else {
550+
return "planned pointer paths require at least two samples"
551+
}
552+
for sample in pointer.samples where sample.point.x != start.x || sample.point.y != start.y {
553+
return "planned tap contacts must not move"
554+
}
555+
if starts.contains(where: { $0.x == start.x && $0.y == start.y }) {
556+
return "planned tap contacts must start on distinct points"
557+
}
558+
starts.append((x: start.x, y: start.y))
559+
}
560+
return nil
561+
}
562+
537563
func plannedGestureExecution(for plan: RunnerGesturePlan) -> PlannedGestureExecution {
538564
plan.topology == "single" && plan.executionProfile == "endpoint-hold"
539565
? .fastSwipe
@@ -582,17 +608,17 @@ extension RunnerTests {
582608
return .performed
583609
#elseif os(tvOS)
584610
return .unsupported(
585-
message: "two-finger gestures are not supported on tvOS",
611+
message: "multi-touch gestures are not supported on tvOS",
586612
hint: "tvOS has no touch input; use remote-driven navigation."
587613
)
588614
#elseif os(visionOS)
589615
return .unsupported(
590-
message: "two-finger touch gestures are not supported on visionOS",
616+
message: "multi-touch gestures are not supported on visionOS",
591617
hint: "The current XCTest synthesizer supports iOS and iPadOS touch simulators only."
592618
)
593619
#else
594620
return .unsupported(
595-
message: "two-finger gestures are not supported on macOS",
621+
message: "multi-touch gestures are not supported on macOS",
596622
hint: "macOS automation has no multi-touch input; run on an iOS simulator."
597623
)
598624
#endif

0 commit comments

Comments
 (0)