Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Conversation

@cbracken
Copy link
Member

@cbracken cbracken commented Jan 22, 2021

This updates the Win32 desktop embedder to support input method (abbreviated IM
or IME) composing regions.

In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.

To illustrate this, in Japanese, this sequence might look something like
the following:

  1. User types 'k'. The character 'k' is added to the composing region.
    Typically, the text 'k' will be inserted inline into the underlying
    text field but the composing range will be highlighted in some manner,
    frequently with a highlight or underline.
  2. User types 'a'. The composing range is replaced with the phonetic
    kana character 'か' (ka). The composing range continues to be
    highlighted.
  3. User types 'k'. The character 'k' is appended to the composing
    range such that the highlighted text is now 'かk'
  4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
    character 'く' (ku) such that the composing range now reads 'かく'
    The composing range continues to be highlighted.
  5. The user presses the space bar to convert the kana characters to
    kanji. The composing range is replaced with '書く' (kaku: to write).
  6. The user presses the space bar again to show other conversions. The
    user's configured input method (for example, Google Japanese Input)
    pops up a completions menu populated with alternatives such as
    各 (kaku: every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus),
    角 (kaku: angle), etc.
  7. The user uses the arrow keys to navigate the completions menu and
    select the alternative to input. As they do, the inline composing
    region in the text field is updated. It continues to be highlighted
    or underlined.
  8. The user hits enter to commit the composing region. The text is
    committed to the underlying text field and the visual highlighting is
    removed.
  9. If the user presses another key, a new composing sequence begins.

If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.

During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, editing_range and text_range. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to text_.length()
exclusive.

Windows IME support revolves around two main UI windows: the composition window
and the candidate window. The composition window is a system window overlaid
within the current window bounds which renders the composing string. Flutter
already renders this string itself, so we request that this window be hidden.
The candidate window is a system-rendered dropdown that displays all possible
conversions for the text in the composing region. Since the contents of this
window are specific to the particular IME in use, and because the user may have
installed one or more third-party IMEs, Flutter does not attempt to render this
as a widget itself, but rather delegates to the system-rendered window.

The lifecycle of IME composing begins follows the following event order:

  1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the
    ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to
    DefWindowProc() in order to hide the composition window, which Flutter
    already renders itself.
  2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new
    text. We use this event to set Flutter's TextInputModel into composing mode.
  3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces,
    or deletes text in the composing region, navigates with their cursor within
    the composing region, or selects a new conversion candidate from the
    candidates list.
  4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text
    in the composing region and decides to commit or abort the composition.

Additionally, the following IME-related events are emitted but not yet handled:

  • WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using
    the system language selection menu. Since there some language-specific
    behaviours to IMEs, we may want to make use of this in the future.
  • WM_IME_NOTIFY: triggered to notify of various status events such as opening
    or closing the candidate window, setting the conversion mode, etc. None of
    these are relevant to Flutter at the moment.
  • WM_IME_REQUEST: triggered to notify of various commands/requests such as
    triggering reconversion of text, which should begin composition mode, insert
    the selected text into the composing region, and allow the user to select new
    alternative candidates for the text in question before re-committing their
    new selection. This patch doesn't support this feature, but it's an important
    feature that we should support in future.

Issues

This is the second in a series of patches addressing flutter/flutter#65574: Full IME support for Windows. It is stacked on top of #23795 (the first commit in this PR, which can be ignored for review purposes).

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
  • I listed at least one issue that this PR fixes in the description above.
  • I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test exempt. See testing the engine for instructions on
    writing and running engine tests.
  • I updated/added relevant documentation (doc comments with ///).
  • I signed the CLA.
  • All existing and new tests are passing.
  • The reviewer has submitted any presubmit flakes in this PR using the engine presubmit flakes form before re-triggering the failure.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@google-cla

This comment has been minimized.

1 similar comment
@googlebot

This comment has been minimized.

@google-cla
Copy link

google-cla bot commented Jan 22, 2021

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

1 similar comment
@googlebot
Copy link

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added the cla: yes label Jan 22, 2021
@cbracken
Copy link
Member Author

@stuartmorgan a couple things before you take a look:

  1. Only the most recent commit needs reviewing as part of this patch -- the rest are from Notify Win32FlutterWindow of cursor updates #23795 and can be reviewed there so this stays a little more focused on handling the events coming from Win32.
  2. I could probably break out the TextInputModel changes into their own patch + tests to shrink this patch down a bit more.

Copy link
Contributor

@stuartmorgan-g stuartmorgan-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with some small changes.

It would be good to look at unit testing the WM handling logic in a follow-up.

void TextInputManager::UpdateCaretRect(HWND hwnd, const Rect& rect) {
caret_rect_ = rect;

// TODO(cbracken): wrap these in an RAII container.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big +1 on this.

This updates the Win32 desktop embedder to support input method (abbreviated IM
or IME) composing regions.

In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.

To illustrate this, in Japanese, this sequence might look something like
the following:

1. User types 'k'. The character 'k' is added to the composing region.
   Typically, the text 'k' will be inserted inline into the underlying
   text field but the composing range will be highlighted in some manner,
   frequently with a highlight or underline.
2. User types 'a'. The composing range is replaced with the phonetic
   kana character 'か' (ka). The composing range continues to be
   highlighted.
3. User types 'k'. The character 'k' is appended to the composing
   range such that the highlighted text is now 'かk'
4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
   character 'く' (ku) such that the composing range now reads 'かく'
   The composing range continues to be highlighted.
5. The user presses the space bar to convert the kana characters to
   kanji. The composing range is replaced with '書く' (kaku: to write).
6. The user presses the space bar again to show other conversions. The
   user's configured input method (for example, ibus) pops up a
   completions menu populated with alternatives such as 各 (kaku:
   every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角
   (kaku: angle), etc.
7. The user uses the arrow keys to navigate the completions menu and
   select the alternative to input. As they do, the inline composing
   region in the text field is updated. It continues to be highlighted
   or underlined.
8. The user hits enter to commit the composing region. The text is
   committed to the underlying text field and the visual highlighting is
   removed.
9. If the user presses another key, a new composing sequence begins.

If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.

During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, `editing_range` and `text_range`. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to `text_.length()`
exclusive.

Windows IME support revolves around two main UI windows: the composition window
and the candidate window. The composition window is a system window overlaid
within the current window bounds which renders the composing string. Flutter
already renders this string itself, so we request that this window be hidden.
The candidate window is a system-rendered dropdown that displays all possible
conversions for the text in the composing region.  Since the contents of this
window are specific to the particular IME in use, and because the user may have
installed one or more third-party IMEs, Flutter does not attempt to render this
as a widget itself, but rather delegates to the system-rendered window.

The lifecycle of IME composing begins follows the following event order:
1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the
   ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to
   DefWindowProc() in order to hide the composition window, which Flutter
   already renders itself.
2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new
   text. We use this event to set Flutter's TextInputModel into composing mode.
3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces,
   or deletes text in the composing region, navigates with their cursor within
   the composing region, or selects a new conversion candidate from the
   candidates list.
4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text
   in the composing region and decides to commit or abort the composition.

Additionally, the following IME-related events are emitted but not yet handled:
* WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using
  the system language selection menu. Since there some language-specific
  behaviours to IMEs, we may want to make use of this in the future.
* WM_IME_NOTIFY: triggered to notify of various status events such as opening
  or closing the candidate window, setting the conversion mode, etc. None of
  these are relevant to Flutter at the moment.
* WM_IME_REQUEST: triggered to notify of various commands/requests such as
  triggering reconversion of text, which should begin composition mode, insert
  the selected text into the composing region, and allow the user to select new
  alternative candidates for the text in question before re-committing their
  new selection. This patch doesn't support this feature, but it's an important
  feature that we should support in future.
@cbracken cbracken merged commit 9365230 into flutter:master Jan 24, 2021
@cbracken cbracken deleted the windows-ime-support branch January 24, 2021 20:56
@cbracken
Copy link
Member Author

LUCI presubmits passed a bunch of times, but on my last push (just cleaning some docs and a little formatting cleanup, they didn't run for some reason :/ Keeping an eye on this post-submit and will revert/reland if I tripped up the formatter.

@chinmaygarde
Copy link
Member

Did you file a flake report at go/flutter-engine-presubmit-flake?

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jan 25, 2021
zanderso pushed a commit to flutter/flutter that referenced this pull request Jan 25, 2021
* 63b3440 Roll Fuchsia Linux SDK from mODEe2CNk... to edqShE0QE... (flutter/engine#23873)

* f0e25c5 Roll Skia from 3193ff271628 to 2a4c0fbdca1a (3 revisions) (flutter/engine#23875)

* 492759e Roll Dart SDK from 82b4c77fb17f to 748993c3997a (1 revision) (flutter/engine#23874)

* 8671aef Notify Win32FlutterWindow of cursor updates (flutter/engine#23795)

* c8620c3 Implement delayed key event synthesis for Windows (flutter/engine#23524)

* ebbf0df Roll Skia from 2a4c0fbdca1a to 8a42b09c162e (9 revisions) (flutter/engine#23878)

* bb00cb6 Roll Fuchsia Linux Toolchain from IJxh_9dNS... to 8LaTdqf7w... (flutter/engine#23876)

* f77fea2 Roll Dart SDK from 748993c3997a to 2ddf810f71f6 (1 revision) (flutter/engine#23881)

* dc22ede Roll Skia from 8a42b09c162e to 9702fc6f3852 (1 revision) (flutter/engine#23882)

* 1f30e56 Roll Fuchsia Mac SDK from tuJCioUf3... to 9Lh_vPIXU... (flutter/engine#23883)

* 443bf5c Roll Fuchsia Linux SDK from edqShE0QE... to uMOnDLfvl... (flutter/engine#23886)

* a152470 Roll Fuchsia Mac SDK from 9Lh_vPIXU... to PsYsfVNbW... (flutter/engine#23888)

* 221259b Roll Skia from 9702fc6f3852 to 07c5f52c947d (2 revisions) (flutter/engine#23890)

* 381d8bd Roll Skia from 07c5f52c947d to 8d29ab630996 (1 revision) (flutter/engine#23892)

* 397274f Roll Skia from 8d29ab630996 to d396cd50ff15 (1 revision) (flutter/engine#23893)

* a5c305e push methods return layers with correct class names (flutter/engine#23542)

* 4e87f60 Read loading unit mapping from AndroidManifest instead of strings (flutter/engine#23868)

* d3a1acb Roll Fuchsia Linux SDK from uMOnDLfvl... to VYUnZ3Tbh... (flutter/engine#23894)

* 3d966fa Roll Fuchsia Mac SDK from PsYsfVNbW... to 6swTf93jz... (flutter/engine#23897)

* cae9130 Roll Skia from d396cd50ff15 to 5bbf72757349 (2 revisions) (flutter/engine#23898)

* f3c5687 Roll Skia from 5bbf72757349 to 069e484cc3b9 (2 revisions) (flutter/engine#23900)

* 9365230 Add support for IME-based text input on Windows (flutter/engine#23853)

* cad597f Roll Fuchsia Linux SDK from VYUnZ3Tbh... to mrFdelzNr... (flutter/engine#23903)

* 4557989 Roll Fuchsia Mac SDK from 6swTf93jz... to 7LGbVIHUD... (flutter/engine#23904)
hjfreyer pushed a commit to hjfreyer/engine that referenced this pull request Mar 22, 2021
This updates the Win32 desktop embedder to support input method (abbreviated IM
or IME) composing regions.

In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.

To illustrate this, in Japanese, this sequence might look something like
the following:

1. User types 'k'. The character 'k' is added to the composing region.
   Typically, the text 'k' will be inserted inline into the underlying
   text field but the composing range will be highlighted in some manner,
   frequently with a highlight or underline.
2. User types 'a'. The composing range is replaced with the phonetic
   kana character 'か' (ka). The composing range continues to be
   highlighted.
3. User types 'k'. The character 'k' is appended to the composing
   range such that the highlighted text is now 'かk'
4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
   character 'く' (ku) such that the composing range now reads 'かく'
   The composing range continues to be highlighted.
5. The user presses the space bar to convert the kana characters to
   kanji. The composing range is replaced with '書く' (kaku: to write).
6. The user presses the space bar again to show other conversions. The
   user's configured input method (for example, ibus) pops up a
   completions menu populated with alternatives such as 各 (kaku:
   every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角
   (kaku: angle), etc.
7. The user uses the arrow keys to navigate the completions menu and
   select the alternative to input. As they do, the inline composing
   region in the text field is updated. It continues to be highlighted
   or underlined.
8. The user hits enter to commit the composing region. The text is
   committed to the underlying text field and the visual highlighting is
   removed.
9. If the user presses another key, a new composing sequence begins.

If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.

During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, `editing_range` and `text_range`. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to `text_.length()`
exclusive.

Windows IME support revolves around two main UI windows: the composition window
and the candidate window. The composition window is a system window overlaid
within the current window bounds which renders the composing string. Flutter
already renders this string itself, so we request that this window be hidden.
The candidate window is a system-rendered dropdown that displays all possible
conversions for the text in the composing region.  Since the contents of this
window are specific to the particular IME in use, and because the user may have
installed one or more third-party IMEs, Flutter does not attempt to render this
as a widget itself, but rather delegates to the system-rendered window.

The lifecycle of IME composing begins follows the following event order:
1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the
   ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to
   DefWindowProc() in order to hide the composition window, which Flutter
   already renders itself.
2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new
   text. We use this event to set Flutter's TextInputModel into composing mode.
3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces,
   or deletes text in the composing region, navigates with their cursor within
   the composing region, or selects a new conversion candidate from the
   candidates list.
4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text
   in the composing region and decides to commit or abort the composition.

Additionally, the following IME-related events are emitted but not yet handled:
* WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using
  the system language selection menu. Since there some language-specific
  behaviours to IMEs, we may want to make use of this in the future.
* WM_IME_NOTIFY: triggered to notify of various status events such as opening
  or closing the candidate window, setting the conversion mode, etc. None of
  these are relevant to Flutter at the moment.
* WM_IME_REQUEST: triggered to notify of various commands/requests such as
  triggering reconversion of text, which should begin composition mode, insert
  the selected text into the composing region, and allow the user to select new
  alternative candidates for the text in question before re-committing their
  new selection. This patch doesn't support this feature, but it's an important
  feature that we should support in future.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants