Skip to content

Reliable initialization of keyboard and pointer bindings  #103094

@dkwingsmt

Description

@dkwingsmt

We should review the messaging mechanism of various bindings (keyboard and pointer) and make them reliable. The current process, where the embedding blindly sends events to the framework, might cause message loss at initialization and state instability, especially during hot restarts.

Analysis

Take keyboard as an example. Currently, when the app is started,

  1. The Flutter view creates both a keyboard manager and a dart VM.
  2. The keyboard manager immediately starts listening to system key events and send messages through the embedder API, FlutterSendKeyEvent.
  3. It takes a short while for the dart VM to boot and run through all static statements, such as assigning a listener to the framework API onKeyData.
  4. If a key event happens between the creation of the keyboard manager and the initialization of the framework, since the listener is still null, the message is lost.

Since the keyboard system only sends events, not the entire state, packet losses are critical and lead to state instability.

Now, we have managed to alleviate this issue by making the keyboard system send messages through the binary messenger to make use of the message buffer (flutter/engine#29795). But the message buffer only has a depth of 1, and it's not difficult to reproduce a case that overflows the buffer (#89748). We can increase the buffer, but that only makes the problem "harder" to occur, not impossible.

The same problem is also found in the pointer event system (#102453 (comment)). We decided to temporarily alleviate this problem by relaxing the assertions.

Moreover, if the framework decides to reconstruct the keyboard manager or pointer manager, there is no way to synchronize the states.

Proposal

There are two options to fix this issue.

  1. Send states, not deltas. By always sending the entire keyboard/pointer state on every change, state stability is guaranteed, and the 1-depth message buffer can be used as intended. However these states can get really big. The user might smash the keyboard, pressing 30 keys at the same time, which becomes 60 uint64s per message, and the framework also needs to compare the state difference to figure out events.
  2. Initiate messaging/synchronization by framework. The framework will send an establishment message when it first assigns a listener, and the engine will reply this message with a full state. This method complicates the API protocol, but keeps the followup messages light.

Both of the two options will possibly be a breaking change to the current embedder API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work lista: desktopRunning on desktopa: mouseIssues related to using a mouse or mouse supporta: text inputEntering text in a text field or keyboard related problemsteam-text-inputOwned by Text Input teamtriaged-text-inputTriaged by Text Input team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions