Skip to content

Commit 9e619e1

Browse files
Add real WebDriver pointer actions to package:integration_test on the web (flutter#195207)
Adds a first-class way for Flutter web integration tests to drive the *real* browser mouse through WebDriver: clicks, hovers, drags, and arbitrary pointer action sequences with any mouse button. `WidgetTester.tap`, `WidgetTester.drag` and `binding.handlePointerEvent` inject synthetic events straight into the framework. They skip the browser's own DOM hit testing and event dispatch, which is what decides whether a pointer event lands on Flutter content or on a platform view (`<iframe>`, map, video) underneath it. So features that depend on browser hit testing, such as pointer handling over platform views, couldn't be checked end to end. Until now the only workaround was to misuse `binding.takeScreenshot` as an RPC and drive the mouse from inside the host's `onScreenshot` callback. ### New API On `IntegrationTestWidgetsFlutterBinding`: - `Future<void> webDriverPointerActions(List<WebDriverPointerAction> actions, {FlutterView? view})` performs `actions` in order as one real browser pointer sequence and completes once the browser has dispatched every event in it. - `Future<void> webDriverClick(Offset position, {FlutterView? view})` is a convenience wrapper that moves to `position` and presses and releases the primary button. In `package:integration_test/common.dart` (which `integration_test.dart` does not re-export, so tests import it directly): - `sealed class WebDriverPointerAction`, created with `WebDriverPointerAction.move(x, y, {Duration duration})`, `.down({WebDriverMouseButton button})`, `.up({WebDriverMouseButton button})` and `.pause(Duration duration)`. The concrete classes are `WebDriverPointerMove`, `WebDriverPointerDown`, `WebDriverPointerUp` and `WebDriverPointerPause`, with `toJson`/`fromJson`, value equality and readable `toString`. - `enum WebDriverMouseButton { primary, middle, secondary }`, whose `w3cButton` is the W3C button index (`0`, `1`, `2`), not a Flutter `kPrimaryButton`-style bit flag. - `WebDriverCommandType.pointerActions` and `WebDriverCommand.pointerActions(actions)` in the web command protocol. - `CallbackManager.pointerActions(List<WebDriverPointerAction> actions)`, which completes with an `UnsupportedError` by default. For example, a drag split across two calls so the test can assert on the intermediate state: ```dart final Offset start = tester.getCenter(find.byKey(dragHandleKey)); await binding.webDriverPointerActions(<WebDriverPointerAction>[ WebDriverPointerAction.move(start.dx, start.dy), const WebDriverPointerAction.down(), WebDriverPointerAction.move(start.dx + 100, start.dy, duration: const Duration(milliseconds: 300)), ]); // ...assert mid-drag... await binding.webDriverPointerActions(<WebDriverPointerAction>[ const WebDriverPointerAction.up(), ]); ``` ### How it works - Move positions are logical positions in the target view (for example from `tester.getCenter`). `webDriverPointerActions` converts them to viewport CSS pixels by adding the view host element's `getBoundingClientRect()` origin, so full-page, embedded and multi-view apps all work. The command travels over the same handshake as `screenshot`. - `integrationDriver` (in `integration_test_driver_extended.dart`) sends the whole sequence as a single W3C perform-actions request for one mouse input source and replies only after WebDriver has performed it. Tests can assert right after the `await` with no fixed sleeps, and no `onScreenshot` callback is needed. - ChromeDriver performs a timed `pointerMove` as a single `mousemove` at the end. To make the browser see intermediate events during drags and hovers, the driver splits each timed move into straight-line steps of about 16 ms, starting from the previous move's position (remembered across calls within one `integrationDriver` run) and capped at one step per pixel. Positions are rounded to whole CSS pixels. - Pressed buttons stay pressed across calls until released with `up`. - Invalid input fails fast in the browser: an empty sequence, non-finite move coordinates, or negative durations throw `ArgumentError` before anything is sent, and calling without a `view` when there is no implicit view throws `StateError`. Off the web, both methods throw `UnsupportedError`. - If the driver isn't a `WebFlutterDriver`, the host fails the in-browser call and throws a `StateError`. If WebDriver throws (for example, coordinates out of bounds), the host releases pressed buttons, fails the in-browser call, and rethrows the original error, so the test fails at once instead of hanging. - Neither method changes `binding.shouldPropagateDevicePointerEvents`. Real browser events reach the test binding as device events, so a test that asserts on Flutter widget gestures has to opt in with `binding.shouldPropagateDevicePointerEvents = true`. Platform view DOM listeners get the native events either way. ### Tests - New web e2e `webdriver_pointer_actions_integration`, registered in `run_web_tests.dart` for CanvasKit and `--wasm` in profile mode. It checks that a real click reaches a same-origin `<iframe>` platform view (which `tester.tap` can't do), that a click reaches a Flutter button only once device pointer events are opted in, that position conversion still works when the view's host element is offset in the page, that a timed drag produces intermediate pointer moves, that buttons stay pressed across calls for a split drag, and that moving without a button hovers. - New unit tests in `packages/integration_test/test/webdriver_command_test.dart` cover action serialization, equality and `fromJson` validation, the `pointerActions` command, the `CallbackManager` default, input validation, and the off-web `UnsupportedError`. - New unit tests in `packages/integration_test/test/integration_driver_pointer_actions_test.dart` cover the W3C request `integrationDriver` builds (rounding, timed-move subdivision across requests, pauses and buttons) and its error handling for non-web drivers, WebDriver failures and malformed action JSON. ### Compatibility `CallbackManager.pointerActions` has a default body, so classes that `extend` `CallbackManager` keep compiling. A class that `implements` it would need to add `pointerActions`. The binding's `callbackManager` can't be swapped out, though, so outside code shouldn't have a real implementation. `common.dart` is not re-exported from `integration_test.dart`, so no new names enter the scope of existing tests. Part of flutter#179806 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 11e802e commit 9e619e1

11 files changed

Lines changed: 2039 additions & 6 deletions

‎dev/bots/suite_runners/run_web_tests.dart‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,17 @@ class WebTestsSuite {
163163
() => _runWebE2eTest('deferred_loading_integration', buildMode: 'release', useWasm: false),
164164
() => _runWebE2eTest('deferred_loading_integration', buildMode: 'release', useWasm: true),
165165

166+
() => _runWebE2eTest(
167+
'webdriver_pointer_actions_integration',
168+
buildMode: 'profile',
169+
useWasm: false,
170+
),
171+
() => _runWebE2eTest(
172+
'webdriver_pointer_actions_integration',
173+
buildMode: 'profile',
174+
useWasm: true,
175+
),
176+
166177
() => _runWebTreeshakeTest(),
167178

168179
() => _runFlutterDriverWebTest(
Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
// Copyright 2014 The Flutter Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:async';
6+
import 'dart:js_interop';
7+
import 'dart:ui_web' as ui_web;
8+
9+
import 'package:flutter/material.dart';
10+
import 'package:web/web.dart' as web;
11+
12+
/// Platform view type of the same-origin `<iframe srcdoc>` that counts clicks
13+
/// and mouse moves.
14+
const String iframeViewType = 'webdriver-pointer-actions-iframe';
15+
16+
/// Key of the Flutter button that counts clicks.
17+
const Key buttonKey = Key('webdriver-click-button');
18+
19+
/// Key of the Flutter pan area that records pan gestures.
20+
const Key panAreaKey = Key('webdriver-pan-area');
21+
22+
/// Key of the Flutter hover box that counts pointer enter and exit events.
23+
const Key hoverKey = Key('webdriver-hover-box');
24+
25+
/// Number of clicks the Flutter button received.
26+
final ValueNotifier<int> buttonClicks = ValueNotifier<int>(0);
27+
28+
/// Number of `click` events the iframe document received.
29+
int iframeClicks = 0;
30+
31+
/// Number of `mousemove` events the iframe document received.
32+
int iframeMouseMoves = 0;
33+
34+
/// Number of `onPanStart` callbacks the pan area received.
35+
int panStarts = 0;
36+
37+
/// Global positions recorded by the pan area's `onPanUpdate`.
38+
final List<Offset> panUpdates = <Offset>[];
39+
40+
/// Number of `onPanEnd` callbacks the pan area received.
41+
int panEnds = 0;
42+
43+
/// Number of `onEnter` callbacks the hover box received.
44+
int hoverEnters = 0;
45+
46+
/// Number of `onExit` callbacks the hover box received.
47+
int hoverExits = 0;
48+
49+
/// Completes once the current iframe document has loaded and registered its
50+
/// listeners.
51+
Completer<void> iframeLoaded = Completer<void>();
52+
53+
bool _factoryRegistered = false;
54+
55+
void main() {
56+
// Each test calls main, so reset the counters and register the view factory
57+
// only once.
58+
buttonClicks.value = 0;
59+
iframeClicks = 0;
60+
iframeMouseMoves = 0;
61+
panStarts = 0;
62+
panUpdates.clear();
63+
panEnds = 0;
64+
hoverEnters = 0;
65+
hoverExits = 0;
66+
iframeLoaded = Completer<void>();
67+
if (!_factoryRegistered) {
68+
ui_web.platformViewRegistry.registerViewFactory(iframeViewType, _createIframe);
69+
_factoryRegistered = true;
70+
}
71+
runApp(const WebDriverPointerActionsApp());
72+
}
73+
74+
web.HTMLIFrameElement _createIframe(int viewId) {
75+
final iframe = web.HTMLIFrameElement()
76+
..srcdoc = '<body style="margin:0;height:100vh;background:#cde"></body>'.toJS
77+
..style.border = 'none'
78+
..style.width = '100%'
79+
..style.height = '100%';
80+
iframe.addEventListener(
81+
'load',
82+
(web.Event _) {
83+
final web.Document doc = iframe.contentDocument!;
84+
doc.addEventListener(
85+
'click',
86+
(web.Event _) {
87+
iframeClicks += 1;
88+
}.toJS,
89+
);
90+
doc.addEventListener(
91+
'mousemove',
92+
(web.Event _) {
93+
iframeMouseMoves += 1;
94+
}.toJS,
95+
);
96+
if (!iframeLoaded.isCompleted) {
97+
iframeLoaded.complete();
98+
}
99+
}.toJS,
100+
);
101+
return iframe;
102+
}
103+
104+
/// Flutter controls (button, pan area, hover box) in a header above a
105+
/// full-width iframe platform view.
106+
class WebDriverPointerActionsApp extends StatelessWidget {
107+
const WebDriverPointerActionsApp({super.key});
108+
109+
@override
110+
Widget build(BuildContext context) {
111+
return MaterialApp(
112+
home: Scaffold(
113+
body: Column(
114+
crossAxisAlignment: CrossAxisAlignment.start,
115+
children: <Widget>[
116+
SizedBox(
117+
height: 100,
118+
child: Row(
119+
crossAxisAlignment: CrossAxisAlignment.start,
120+
children: <Widget>[
121+
ValueListenableBuilder<int>(
122+
valueListenable: buttonClicks,
123+
builder: (BuildContext context, int clicks, Widget? child) {
124+
return TextButton(
125+
key: buttonKey,
126+
onPressed: () {
127+
buttonClicks.value += 1;
128+
},
129+
child: Text('Button clicks: $clicks'),
130+
);
131+
},
132+
),
133+
const SizedBox(width: 16),
134+
GestureDetector(
135+
key: panAreaKey,
136+
behavior: HitTestBehavior.opaque,
137+
onPanStart: (DragStartDetails _) {
138+
panStarts += 1;
139+
},
140+
onPanUpdate: (DragUpdateDetails details) {
141+
panUpdates.add(details.globalPosition);
142+
},
143+
onPanEnd: (DragEndDetails _) {
144+
panEnds += 1;
145+
},
146+
child: const SizedBox(
147+
width: 300,
148+
height: 60,
149+
child: ColoredBox(color: Color(0xFFE0E0E0)),
150+
),
151+
),
152+
const SizedBox(width: 16),
153+
MouseRegion(
154+
key: hoverKey,
155+
onEnter: (_) {
156+
hoverEnters += 1;
157+
},
158+
onExit: (_) {
159+
hoverExits += 1;
160+
},
161+
child: const SizedBox(
162+
width: 100,
163+
height: 60,
164+
child: ColoredBox(color: Color(0xFFB0D0E0)),
165+
),
166+
),
167+
],
168+
),
169+
),
170+
const Expanded(child: HtmlElementView(viewType: iframeViewType)),
171+
],
172+
),
173+
),
174+
);
175+
}
176+
}

0 commit comments

Comments
 (0)