Skip to content

Commit aa3e869

Browse files
author
Dmitrii Zolotov
committed
Add the mouse_capture plugin: relative pointer deltas on macOS
Flutter exposes no pointer lock on any desktop platform, which makes a first-person camera impossible out of the box: the cursor reaches the edge of the window and the view stops turning. flutter/flutter#134652 is open. The mechanism is the one AppKit intends. Turning off the association between the physical mouse and the on-screen cursor freezes the cursor while mouseMoved events keep arriving with their deltas intact; hide the cursor, read deltaX and deltaY from a local NSEvent monitor, and that is all of it. Warping the cursor back to a fixed point every frame also works but fights the window server and shows as jitter, so it stays a fallback. helgoboss/pointer_lock (MIT, unpublished on pub.dev) already does this across four platforms and is where the technique came from. This is a separate plugin because three of its gaps matter to a game: no observer for focus loss, so Cmd+Tab strands the cursor system-wide; one channel message per mouse event at a 1000 Hz polling rate; and no isSupported, which a build targeting mobile needs. The delta is drained with takeDelta rather than pushed on a stream. That is the shape a fixed-timestep simulation wants — it asks how far the mouse moved since the last step, once per step — and it is synchronous, because awaiting inside the step would mean the step no longer sees a consistent snapshot of its inputs. Two traps, both now covered. A plugin outlives the Dart isolate, so a hot restart while captured leaves the cursor hidden with nothing left to ask for it back; construction issues a reset, which is only safe because the native side keeps its own flag — hide and unhide are counted, and an unbalanced unhide quietly breaks the next hide. And focus loss must release rather than pause, or a hidden cursor ends up over another application's window. A package rather than app code: it knows nothing about the engine or the game and can be lifted out whole. The example app exists to isolate it — capture fails in ways that look exactly like a broken camera, and telling those apart inside a 3D scene means debugging two things at once. 17 tests on the Dart accumulator and state machine, which is where the easy-to-miss failures live. The native side is verified by running the example.
1 parent 4e08fd7 commit aa3e869

54 files changed

Lines changed: 3113 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -233,6 +233,8 @@ either does not start or silently renders nothing. Worth keeping to hand.
233233
| Shadows look right but toggling them changes nothing | Check the setting actually reaches `RenderSettings`. A control wired to the panel but not to the renderer looks completely convincing, and the way to find out is to capture the same frame with the feature on and off and diff the two — a zero difference is the whole answer |
234234
| A draw is submitted, the counter goes up, nothing appears | There is **no non-indexed draw**. `draw()` with only a vertex buffer bound succeeds and renders nothing. Bind an index buffer even when the indices are the identity `0, 1, 2, …` sequence — the debug line overlay keeps one in a device buffer that only grows |
235235
| `PathAccessException … Operation not permitted` when writing a file | macOS Flutter apps are sandboxed. Anything outside `~/Library/Containers/<bundle id>/Data` is refused, so frame captures resolve relative paths against the app's own temp directory |
236+
| The view stops turning when the cursor reaches the edge of the window | Flutter exposes no pointer lock on any desktop platform. `packages/mouse_capture` supplies it on macOS by turning off the association between the physical mouse and the on-screen cursor, which leaves `mouseMoved` events arriving with their deltas while the cursor stays put |
237+
| No cursor anywhere after a hot restart | A plugin holding the pointer outlives the Dart isolate, because the engine's registrar owns it. The Dart side comes back remembering nothing, so nothing asks for the cursor back. `MouseCapture` issues a reset on construction for exactly this |
236238

237239
## Layout
238240

@@ -260,6 +262,7 @@ lib/src/engine/scene/ scene graph, cameras, lights, orbit, raycasting
260262
lib/src/engine/render/ renderer, render list, materials, sorting, debug draw
261263
lib/src/engine/assets/ glTF, OBJ and .f3d decoders, isolate loading, cache
262264
lib/src/spike/ demo glue and the frame capture hook
265+
packages/mouse_capture/ relative mouse deltas; knows nothing about the engine
263266
test/ 424 tests, all runnable without a GPU
264267
```
265268

‎RESEARCH.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -618,6 +618,52 @@ What remains unverified: a benchmark of draw calls and GC pauses at 10k transfor
618618

619619
---
620620

621+
## 6. Pointer capture, which the framework does not provide
622+
623+
Recorded here because it is a gap in Flutter rather than in `flutter_gpu`, and
624+
because a first-person camera is impossible without closing it: the cursor
625+
reaches the edge of the window and the view stops turning. The framework exposes
626+
no pointer lock on **any** desktop platform, and the request for one
627+
([flutter/flutter#134652](https://github.com/flutter/flutter/issues/134652)) is
628+
open.
629+
630+
**What works on macOS.** Turning off the association between the physical mouse
631+
and the on-screen cursor —
632+
`CGAssociateMouseAndMouseCursorPosition(0)` — freezes the cursor while
633+
`mouseMoved` events keep arriving with their deltas intact. Hide the cursor with
634+
`NSCursor.hide()`, read `event.deltaX/deltaY` from a local `NSEvent` monitor, and
635+
that is the whole mechanism. Warping the cursor back to a fixed point every frame
636+
also works, but it fights the window server and shows as jitter, so it is a
637+
fallback and not the design.
638+
639+
This was not guesswork: [helgoboss/pointer_lock](https://github.com/helgoboss/pointer_lock)
640+
(MIT, unpublished on pub.dev) already does it this way across four platforms, and
641+
is where the technique came from. Our own `packages/mouse_capture` exists because
642+
three of its gaps matter to a game — no observer for focus loss, so Cmd+Tab
643+
strands the cursor system-wide; one platform-channel message per mouse event at a
644+
1000 Hz polling rate; and no `isSupported`, which a build targeting mobile needs.
645+
646+
**Two traps worth naming.**
647+
648+
*The cursor outlives the Dart isolate.* A plugin is owned by the engine's
649+
registrar, so a hot restart while the pointer is captured leaves the native side
650+
still holding it with nothing left that remembers to ask for it back — no cursor,
651+
and no way to recover except quitting. The fix is a `reset` issued on
652+
construction, and it only works because the native side keeps its own flag: hide
653+
and unhide are counted, so unhiding without having hidden would leave the count
654+
negative and quietly break the next hide.
655+
656+
*Focus loss must release, not pause.* Anything else leaves a hidden cursor over
657+
another application's window. Both `NSApplication.didResignActive` and
658+
`NSWindow.didResignKey` are needed — the first fires on Cmd+Tab, the second when
659+
another window of the same app takes focus.
660+
661+
**Not yet measured**: whether one channel message per mouse event is visible in a
662+
frame profile. If it is, the fix is to accumulate natively and flush once per
663+
frame.
664+
665+
---
666+
621667
## Sources
622668

623669
- [flutter_gpu library — Dart API](https://api.flutter.dev/flutter/flutter_gpu/)
@@ -634,3 +680,5 @@ What remains unverified: a benchmark of draw calls and GC pauses at 10k transfor
634680
- [Clustered Lighting — PlayCanvas Developer Site](https://developer.playcanvas.com/user-manual/graphics/lighting/clustered-lighting/)
635681
- [PlayCanvas Engine](https://playcanvas.com/products/engine)
636682
- [[Impeller] Support compute passes/shaders — flutter/flutter#109346](https://github.com/flutter/flutter/issues/109346)
683+
- [Flutter should support capturing the mouse pointer — flutter/flutter#134652](https://github.com/flutter/flutter/issues/134652)
684+
- [helgoboss/pointer_lock — GitHub](https://github.com/helgoboss/pointer_lock)

‎macos/Flutter/GeneratedPluginRegistrant.swift‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
import FlutterMacOS
66
import Foundation
77

8+
import mouse_capture
89

910
func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) {
11+
MouseCapturePlugin.register(with: registry.registrar(forPlugin: "MouseCapturePlugin"))
1012
}

‎packages/mouse_capture/.gitignore‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Miscellaneous
2+
*.class
3+
*.log
4+
*.pyc
5+
*.swp
6+
.DS_Store
7+
.atom/
8+
.build/
9+
.buildlog/
10+
.history
11+
.svn/
12+
.swiftpm/
13+
migrate_working_dir/
14+
15+
# IntelliJ related
16+
*.iml
17+
*.ipr
18+
*.iws
19+
.idea/
20+
21+
# The .vscode folder contains launch configuration and tasks you configure in
22+
# VS Code which you may wish to be included in version control, so this line
23+
# is commented out by default.
24+
#.vscode/
25+
26+
# Flutter/Dart/Pub related
27+
# Libraries should not include pubspec.lock, per https://dart.dev/guides/libraries/private-files#pubspeclock.
28+
/pubspec.lock
29+
**/doc/api/
30+
.dart_tool/
31+
.flutter-plugins-dependencies
32+
/build/
33+
/coverage/

‎packages/mouse_capture/.metadata‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# This file tracks properties of this Flutter project.
2+
# Used by Flutter tool to assess capabilities and perform upgrades etc.
3+
#
4+
# This file should be version controlled and should not be manually edited.
5+
6+
version:
7+
revision: "ee80f08bbf97172ec030b8751ceab557177a34a6"
8+
channel: "stable"
9+
10+
project_type: plugin
11+
12+
# Tracks metadata for the flutter migrate command
13+
migration:
14+
platforms:
15+
- platform: root
16+
create_revision: ee80f08bbf97172ec030b8751ceab557177a34a6
17+
base_revision: ee80f08bbf97172ec030b8751ceab557177a34a6
18+
- platform: macos
19+
create_revision: ee80f08bbf97172ec030b8751ceab557177a34a6
20+
base_revision: ee80f08bbf97172ec030b8751ceab557177a34a6
21+
22+
# User provided section
23+
24+
# List of Local paths (relative to this file) that should be
25+
# ignored by the migrate tool.
26+
#
27+
# Files that are not part of the templates will be ignored by default.
28+
unmanaged_files:
29+
- 'lib/main.dart'
30+
- 'ios/Runner.xcodeproj/project.pbxproj'
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## 0.1.0
2+
3+
* Pointer capture on macOS: the cursor is frozen and hidden while motion arrives
4+
as relative deltas.
5+
* `takeDelta` drains an accumulator, which is the shape a fixed-timestep
6+
simulation wants.
7+
* Capture is released when the window or the application loses focus, and the
8+
release is announced on `onStateChanged`.
9+
* Construction resets the native side, so a hot restart cannot strand the cursor.
10+
* `isSupported` answers honestly on platforms with no implementation instead of
11+
throwing at the first call.

‎packages/mouse_capture/LICENSE‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
TODO: Add your license here.

‎packages/mouse_capture/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# mouse_capture
2+
3+
Locks the mouse pointer in place and reports its motion as relative deltas.
4+
5+
Flutter exposes no pointer lock on any desktop platform, which makes a
6+
first-person camera impossible out of the box: the cursor reaches the edge of the
7+
window and the view stops turning. This plugin is the missing piece.
8+
9+
```dart
10+
final capture = MouseCapture.instance;
11+
12+
if (capture.isSupported) {
13+
await capture.capture();
14+
}
15+
16+
// Once per simulation step:
17+
final delta = capture.takeDelta();
18+
yaw += delta.dx * sensitivity;
19+
pitch += delta.dy * sensitivity;
20+
```
21+
22+
## Pull, not push
23+
24+
The delta is drained with `takeDelta` rather than delivered by a stream, because
25+
that is the shape a game wants. A simulation running on a fixed timestep asks
26+
"how far did the mouse move since the last step" once per step; handing it a
27+
stream of individual mouse events only moves the accumulation into every caller,
28+
and each of them has to get it right.
29+
30+
`takeDelta` is synchronous for the same reason — it is called from inside the
31+
step, where awaiting anything would mean the step no longer sees a consistent
32+
snapshot of its inputs.
33+
34+
## Releasing
35+
36+
Losing window focus drops the capture and announces it on `onStateChanged`.
37+
Anything else would leave a hidden cursor over another application, which the
38+
user cannot recover from. Treat an unrequested `CaptureState.released` as a
39+
reason to pause.
40+
41+
The plugin does **not** watch for Escape. Key handling belongs to the
42+
application, which calls `release()` itself.
43+
44+
## Platforms
45+
46+
| Platform | Status | How |
47+
|---|---|---|
48+
| macOS | supported | `CGAssociateMouseAndMouseCursorPosition(0)`, `NSCursor.hide()`, a local `NSEvent` monitor |
49+
| Windows | not yet | Raw Input plus `ClipCursor` |
50+
| Linux | not yet | `gdk_seat_grab`, or XI2 raw events |
51+
| iOS, Android | not applicable | no pointer to capture; `isSupported` is false |
52+
53+
`isSupported` is a whitelist rather than a try-and-see, so an application can
54+
offer another control scheme instead of discovering the gap at the first call.
55+
56+
## Hot restart
57+
58+
The native side outlives the Dart isolate. A hot restart while the pointer is
59+
captured would otherwise leave the cursor hidden with nothing left that
60+
remembers to ask for it back, so construction always issues a reset first.
61+
62+
## Prior art
63+
64+
[helgoboss/pointer_lock](https://github.com/helgoboss/pointer_lock) (MIT) solves
65+
the same problem across more platforms and is where the macOS technique here came
66+
from. It is not published on pub.dev. This plugin exists because three things it
67+
does not do matter to a game:
68+
69+
- no observer for focus loss, so Cmd+Tab strands the cursor system-wide;
70+
- one channel message per mouse event, at a 1000 Hz polling rate;
71+
- no `isSupported`, which a build targeting mobile needs.
72+
73+
## Cost
74+
75+
One platform-channel message per mouse event, carrying a two-element
76+
`Float64List`. Whether that shows up in a frame profile at a 1000 Hz polling rate
77+
has not been measured yet; if it does, the fix is to accumulate natively and
78+
flush once per frame.
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
include: package:flutter_lints/flutter.yaml
2+
3+
# Additional information about this file can be found at
4+
# https://dart.dev/guides/language/analysis-options
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Miscellaneous
2+
*.class
3+
*.log
4+
*.pyc
5+
*.swp
6+
.DS_Store
7+
.atom/
8+
.build/
9+
.buildlog/
10+
.history
11+
.svn/
12+
.swiftpm/
13+
migrate_working_dir/
14+
15+
# IntelliJ related
16+
*.iml
17+
*.ipr
18+
*.iws
19+
.idea/
20+
21+
# The .vscode folder contains launch configuration and tasks you configure in
22+
# VS Code which you may wish to be included in version control, so this line
23+
# is commented out by default.
24+
#.vscode/
25+
26+
# Flutter/Dart/Pub related
27+
**/doc/api/
28+
**/ios/Flutter/.last_build_id
29+
.dart_tool/
30+
.flutter-plugins-dependencies
31+
.pub-cache/
32+
.pub/
33+
/build/
34+
/coverage/
35+
36+
# Symbolication related
37+
app.*.symbols
38+
39+
# Obfuscation related
40+
app.*.map.json
41+
42+
# Android Studio will place build artifacts here
43+
/android/app/debug
44+
/android/app/profile
45+
/android/app/release

0 commit comments

Comments
 (0)