Conversation
|
Preview available at https://egui-pr-preview.github.io/pr/7967-fix-undesired-keyboard-input-events-during-ime-composition View snapshot changes at kitdiff |
|
Related |
winit::event::KeyboardInput fiddling with IME composition on Windowswinit::event::KeyboardInput fiddling with IME composition on Windows properly
|
Hello. @choyunjin (Issue #7809) |
|
Tested on Windows 11 with Google Japanese IME, it's working very well, like other applications that supports IME. Nice work! Speaking of implementation, as the behavior which winit sends keyboard events that is consumed by IME is confirmed as a bug, it would be nice to have a comment that says the implementation is workaround and will be fixed by winit. |
winit::event::KeyboardInput fiddling with IME composition on Windows properlywinit::event::KeyboardInput fiddling with IME composition on Windows properly
winit::event::KeyboardInput fiddling with IME composition on Windows properlywinit::event::KeyboardInput fiddling with IME composition on Windows properly
lucasmerlin
left a comment
There was a problem hiding this comment.
Looks good to me! Thanks for the thorough testing as well!
|
I noticed that @rustbasic reacted with an 👀 to the comment where I pinged people, so they may still be reviewing this PR. I think we should wait for their feedback before considering merging it. |
|
Great job, @umajho. On a side note, I believe #7914 should still be pursued for separate reasons. While not directly related to this (#7914 and #7967), there are still some unresolved issues regarding IME handling. |
|
@rustbasic thanks for the feedback! Could you elaborate more on the Cheonjiin issue Specifically:
Regarding #4137The I have tested #7983 on Windows and confirmed that the partial fix for the focus-related bug still works there. (And on macOS, the behavior remains incorrect as before.) I guess the ultimate solution may involve explicitly disabling and re-enabling the IME when a |
|
Regarding the I'm also planning to introduce this in a follow-up PR that implements visuals for “real IME cursors & cursor ranges.” My intention is to work on it after this PR and #7983 are resolved. By “real IME cursors & cursor ranges,” I mean something like the following: Since this discussion is already outside the scope of this PR, I suggest we continue it in #7983 and merge this PR first. (Since I've gathered feedback from both a Japanese IME user and a Korean IME user that this PR works, and I'm confident it also works for Chinese IMEs (at least for those Pinyin-like), I believe this PR is ready to be merged.) |
Regarding the Cheonjiin Korean Keyboard issue: Example sequence: Regarding #7983: Regarding disabling and re-enabling IME on focus: |
|
@umajho Update from further testing: |
|
Note I moved the original content of this comment to #7983 (comment) I think we should continue our discussion in #7983, since we are now talking about what comes after this PR, which is what I originally planned for #7983. |
|
Finally, a smooth typing experience in egui on Windows! Thank you for the extensive testing across different environments. You've solved a long-standing pain point for us : ) |
winit::event::KeyboardInput fiddling with IME composition on Windows properly|
Thank you! |
<!-- Please read the "Making a PR" section of [`CONTRIBUTING.md`](https://github.com/emilk/egui/blob/main/CONTRIBUTING.md) before opening a Pull Request! * Keep your PR:s small and focused. * The PR title is what ends up in the changelog, so make it descriptive! * If applicable, add a screenshot or gif. * If it is a non-trivial addition, consider adding a demo for it to `egui_demo_lib`, or a new example. * Do NOT open PR:s from your `master` branch, as that makes it hard for maintainers to test and add commits to your PR. * Remember to run `cargo fmt` and `cargo clippy`. * Open the PR as a draft until you have self-reviewed it and run `./scripts/check.sh`. * When you have addressed a PR comment, mark it as resolved. Please be patient! I will review your PR, but my time is limited! --> * Closes emilk#7809 * Closes emilk#7876 * Closes emilk#7908 * Supersedes emilk#7877 * Supersedes emilk#7898 * The author of the PR above replaced it with emilk#7914, which additionally fixes another IME issue. I believe that fix deserves a separate PR. * Reverts emilk#4794 * [x] I have followed the instructions in the PR template This approach is better than emilk#7898 (emilk#7914) because it correctly handles all three major IME types (Chinese, Japanese, and Korean) without requiring a predefined “IME mode”. ## Environments I haved tested this PR in <details><summary>macOS 15.7.3 (AArch64, Host of other virtual machines)</summary> Run command: `cargo run -p egui_demo_app --release` Tested IMEs: - builtin Chinese IME (Shuangpin - Simplified) - builtin Japanese IME (Romaji) - builtin Korean IME (2-Set) </details> <details><summary>Windows 11 25H2 (AArch64, Virtual Machine)</summary> Build command: `cargo build --release -p egui_demo_app --target=x86_64-pc-windows-gnu --features=glow --no-default-features` (I cannot use `wgpu` due to [this bug](emilk#4381), which prevents debugging inside the VM. Anyways, the rendering backend should be irrelevant here.) Tested IMEs: - builtin Chinese IME (Shuangpin) - Sogou IME (Chinese Shuangpin) - WeType IME (Chinese Shuangpin) - builtin Japanese IME (Hiragana) - builtin Korean IME (2 Beolsik) </details> <details><summary>Linux [Wayland + IBus] (AArch64, Virtual Machine)</summary> Fedora KDE Plasma Desktop 43 [Wayland + IBus 1.5.33-rc2] (Not working at the moment because of [another issue](emilk#7485) that will be fixed by emilk#7983. It is [a complicated story](emilk#7973 (comment)). ) > [!NOTE] > > IBus is partially broken in this system. The Input Method Selector refuses to select IBus. As a workaround, I have to open System Settings -> Virtual Keyboard and select “IBus Wayland” to start an IBus instance that works in egui. > > The funny thing is: the Chinese Intelligent Pinyin IME is broken in native Apps like System Settings and KWrite, but works correctly in egui! > > <details><summary>Screencast: What</summary> > >  > </details> Build command: `cross build --release -p egui_demo_app --target=aarch64-unknown-linux-gnu --features=wayland,wgpu --no-default-features` (The Linux toolchain on my mac is somehow broken, so I used `cross` instead.) Tested IMEs: - Chinese Intelligent Pinyin IME (Shuangpin) - Japanese Anthy IME (Hiragana) - Korean Hangul IME </details> <details><summary>Linux [X11 + Fcitx5] (AArch64, Virtual Machine)</summary> Debian 13 [Cinnamon 6.4.10 + X11 + Fcitx5 5.1.2] Build command: `cross build --release -p egui_demo_app --target=aarch64-unknown-linux-gnu --features=x11,wgpu --no-default-features` Tested IMEs: - Chinese Shuangpin IME - Chinese Rime IME with `luna-pinyin` - Japanese Mozc IME (Hiragana) - Korean Hangul IME Unlike macOS and Linux + Wayland, key-release events for keys processed by the IME are still forwarded to `egui`. These appear to be harmless in practice. Unlike on Windows, however, they cannot be filtered reliably because there are no corresponding key-press events marked as “processed by IME”. </details> --- There are too many possible combinations to test (Operating Systems × [Desktop Environment](https://en.wikipedia.org/wiki/Desktop_environment)s × [Windowing System](https://en.wikipedia.org/wiki/Windowing_system)s × [IMF](https://wiki.archlinux.org/title/Input_method#Input_method_framework)s × [IME](https://en.wikipedia.org/wiki/Input_method)s × …), and I only have access to a limited subset. For example, Google Japanese Input refused to install on my Windows VM, and some paid Japanese IMEs are not accessible to me. Therefore, I would appreciate feedback from people other than me using all kinds of environments. ## Details There are two possible approaches to removing keyboard events that have already been processed by an IME: * Approach 1: Filter out events inside `egui` that appear to have been received during IME composition. * Approach 2: Filter out such events in the platform backend (terminology [borrowed from imgui](https://github.com/ocornut/imgui/blob/master/docs/BACKENDS.md#using-standard-backends), e.g. the `egui-winit` crate or the code under `web/` in the `eframe` crate.). Both approaches already exist in `egui`: * emilk#4794 uses the first approach, filtering these events in the `TextEdit`-related code. * `eframe` uses the second approach in its web integration. See: <https://github.com/emilk/egui/blob/14afefa2521d1baaf4fd02105eec2d3727a7ac36/crates/eframe/src/web/events.rs#L173-L176> Compared to the first approach, the second has a clear advantage: when events are passed from the platform backends into `egui`, they are simplified and lose information. In contrast, events in the platform backends are the original events, which allows them to be handled more flexibly. This is also why emilk#7898 (emilk#7914), which attempts to address the issue from within the `egui` crate, struggles to make all IMEs work correctly at the same time and requires manually selecting an “IME mode”: the events received by `egui` have already been reduced and therefore lack necessary information. A more appropriate solution is to consistently follow the second approach, explicitly requiring platform backends not to forward events that have already been processed by the IME to `egui`. This is the method used in this PR. Specifically, this PR works within the `egui-winit` crate, where the original `KeyboardInput` events can be accessed. At least for key press events, these can be used directly to determine whether the event has already been processed by the IME on Windows (by checking whether `logical_key` equals `winit::keyboard::NamedKey::Process`). This makes it straightforward to ensure that all IMEs work correctly at the same time. This PR also reverts emilk#4794, which took the first approach. It filters out some events that merely look like they were received during IME composition but actually are not. It also messes up the order of those events along the way. As a result, it caused several IME-related issues. One of the sections in the Demonstrations below will illustrate these problems. ## Demonstrations <details><summary>Changes not included in this PR for displaying Unicode characters in demonstrations</summary> Download `unifont-17.0.03.otf` from <https://unifoundry.com/pub/unifont/unifont-17.0.03/font-builds/>, and place it at `crates/egui_demo_app/src/unifont-17.0.03.otf`. In `crates/egui_demo_app/src/wrap_app.rs`, add these lines at the beginning of `impl WrapApp`'s `pub fn new`: ```rust { const MAIN_FONT: &'static [u8] = include_bytes!("./unifont-17.0.03.otf"); let mut fonts = egui::FontDefinitions::default(); fonts.font_data.insert( "main-font".to_owned(), std::sync::Arc::new(egui::FontData::from_static(MAIN_FONT)), ); let proportional = fonts .families .entry(egui::FontFamily::Proportional) .or_default(); proportional.insert(0, "main-font".to_owned()); cc.egui_ctx.set_fonts(fonts); } ``` (I took this from somewhere, but I forgot where it is. Sorry…) </details> [GNU Unifont](https://unifoundry.com/unifont/index.html) is licensed under [OFL-1.1](https://unifoundry.com/OFL-1.1.txt). ### This PR Fixes: Focus on a single-line `TextEdit` is lost after completing candidate selection with Japanese IME on Windows (emilk#7809) <details><summary>Screencast: ✅ Japanese IME now behaves correctly while Korean IME behaves as before</summary>  </details> ### This PR Fixes: Committing Japanese IME text with <kbd>Enter</kbd> inserts an unintended newline in multiline `TextEdit` on Windows (emilk#7876) <details><summary>Screencast: ✅ Japanese IME now behaves correctly while Korean IME behaves as before</summary>  </details> ### This PR Fixes: Backspacing deletes characters during composition in certain Chinese IMEs (e.g., Sogou) on Windows (emilk#7908) <details><summary>Screencast: ✅ Sogou IME now behaves correctly</summary>  </details> ### This PR Obsoletes emilk#4794, because `egui` receives only IME events during composition from now on On Windows, “incompatible” events are filtered in `egui-winit`, aligning the behavior with other systems. <details><summary>Screencasts</summary> Some Chinese IMEs on Windows:  The default Japanese IMEs on Windows:  </details> The 2-set Korean IMEs handle arrow keys differently. It will be discussed in the next section. ### This PR Reverts emilk#4794, because it introduced several bugs Some of its bugs have already been worked around in the past, but those workarounds might also be problematic. For example, emilk#4912 is a workaround for a bug (emilk#4908) introduced by emilk#4794, and that workaround is in fact the root cause of the macOS backspacing bug I have worked around with emilk#7810. (The reversion of emilk#4912 is out of the scope of this PR, I will do that in emilk#7983.) #### It Caused: Arrow keys are incorrectly blocked during typical Korean IME composition When composing Korean text using 2-Set IMEs, users should still be able to move the cursor with arrow keys regardless if the composition is committed. ##### Correct behavior <details><summary>Screencasts</summary> macOS TextEdit:  Windows Notepad:  With emilk#4794 reverted, `egui` also behaves correctly (tested on Linux + Wayland, macOS, and Windows):  </details> ##### Incorrect behavior caused by emilk#4794 `remove_ime_incompatible_events` removed arrow-key events in such cases. As a result, the first arrow key press only commits the composition, and users need to press the arrow key again to move the cursor: <details><summary>Screencast</summary>  </details> This is essentially the same issue described here: emilk#7877 (comment) #### It Caused: Backspacing leaves the last character in Korean IME pre-edit text not removed on macOS <details><summary>Screencasts</summary> Before this PR:  After this PR:  </details> ### Korean IMEs also use <kbd>Enter</kbd> to confirm Hanja selections, and will not work properly in the Korean “IME mode” proposed by emilk#7898 (emilk#7914) <details><summary>Screencast: Korean IME using <kbd>Enter</kbd> and <kbd>Space</kbd> for confirmation (IBus Korean Hangul IME)</summary> The screencast below demonstrates that some Korean IMEs handle Hanja selection in a way similar to Japanese IMEs: the <kbd>Up</kbd>/<kbd>Down</kbd> arrow keys are used to navigate candidates, and <kbd>Enter</kbd> confirms the selected candidate.  </details> <details><summary>Screencasts: Another example</summary> Using the built-in Korean IME on Windows, I type two lines: the first line in Hangul, and the second line as the same word converted to Hanja. Correct behavior in Notepad (reference):  Behavior after applying this PR, which matches the Notepad behavior:  Behavior after applying emilk#7914 with the “IME mode” set to Korean (which is also the behavior before this PR being applied):  On the second line, each time a Hanja character is confirmed, an unintended newline is inserted. This mirrors the Japanese IME issues that are supposed to be fixed by setting the “IME mode” to Japanese. (These Japanese IME issues are fixed in this PR as mentioned before.) </details>


TextEditinput is removed when typing is completed with Japanese IME #7809TextEditto fix some bugs #4794This approach is better than #7898 (#7914) because it correctly handles all three major IME types (Chinese, Japanese, and Korean) without requiring a predefined “IME mode”.
Environments I haved tested this PR in
macOS 15.7.3 (AArch64, Host of other virtual machines)
Run command:
cargo run -p egui_demo_app --releaseTested IMEs:
Windows 11 25H2 (AArch64, Virtual Machine)
Build command:
cargo build --release -p egui_demo_app --target=x86_64-pc-windows-gnu --features=glow --no-default-features(I cannot use
wgpudue to this bug, which prevents debugging inside the VM. Anyways, the rendering backend should be irrelevant here.)Tested IMEs:
Linux [Wayland + IBus] (AArch64, Virtual Machine)
Fedora KDE Plasma Desktop 43 [Wayland + IBus 1.5.33-rc2]
(Not working at the moment because of another issue that will be fixed by #7983. It is a complicated story. )
Build command:
cross build --release -p egui_demo_app --target=aarch64-unknown-linux-gnu --features=wayland,wgpu --no-default-features(The Linux toolchain on my mac is somehow broken, so I used
crossinstead.)Tested IMEs:
Linux [X11 + Fcitx5] (AArch64, Virtual Machine)
Debian 13 [Cinnamon 6.4.10 + X11 + Fcitx5 5.1.2]
Build command:
cross build --release -p egui_demo_app --target=aarch64-unknown-linux-gnu --features=x11,wgpu --no-default-featuresTested IMEs:
luna-pinyinUnlike macOS and Linux + Wayland, key-release events for keys processed by the IME are still forwarded to
egui. These appear to be harmless in practice.Unlike on Windows, however, they cannot be filtered reliably because there are no corresponding key-press events marked as “processed by IME”.
There are too many possible combinations to test (Operating Systems × Desktop Environments × Windowing Systems × IMFs × IMEs × …), and I only have access to a limited subset. For example, Google Japanese Input refused to install on my Windows VM, and some paid Japanese IMEs are not accessible to me. Therefore, I would appreciate feedback from people other than me using all kinds of environments.
Details
There are two possible approaches to removing keyboard events that have already been processed by an IME:
eguithat appear to have been received during IME composition.egui-winitcrate or the code underweb/in theeframecrate.).Both approaches already exist in
egui:TextEditto fix some bugs #4794 uses the first approach, filtering these events in theTextEdit-related code.eframeuses the second approach in its web integration. See:egui/crates/eframe/src/web/events.rs
Lines 173 to 176 in 14afefa
Compared to the first approach, the second has a clear advantage: when events are passed from the platform backends into
egui, they are simplified and lose information. In contrast, events in the platform backends are the original events, which allows them to be handled more flexibly. This is also why #7898 (#7914), which attempts to address the issue from within theeguicrate, struggles to make all IMEs work correctly at the same time and requires manually selecting an “IME mode”: the events received byeguihave already been reduced and therefore lack necessary information.A more appropriate solution is to consistently follow the second approach, explicitly requiring platform backends not to forward events that have already been processed by the IME to
egui. This is the method used in this PR. Specifically, this PR works within theegui-winitcrate, where the originalKeyboardInputevents can be accessed. At least for key press events, these can be used directly to determine whether the event has already been processed by the IME on Windows (by checking whetherlogical_keyequalswinit::keyboard::NamedKey::Process). This makes it straightforward to ensure that all IMEs work correctly at the same time.This PR also reverts #4794, which took the first approach. It filters out some events that merely look like they were received during IME composition but actually are not. It also messes up the order of those events along the way.
As a result, it caused several IME-related issues. One of the sections in the Demonstrations below will illustrate these problems.
Demonstrations
Changes not included in this PR for displaying Unicode characters in demonstrations
Download
unifont-17.0.03.otffrom https://unifoundry.com/pub/unifont/unifont-17.0.03/font-builds/, and place it atcrates/egui_demo_app/src/unifont-17.0.03.otf.In
crates/egui_demo_app/src/wrap_app.rs, add these lines at the beginning ofimpl WrapApp'spub fn new:(I took this from somewhere, but I forgot where it is. Sorry…)
GNU Unifont is licensed under OFL-1.1.
This PR Fixes: Focus on a single-line
TextEditis lost after completing candidate selection with Japanese IME on Windows (#7809)Screencast: ✅ Japanese IME now behaves correctly while Korean IME behaves as before
This PR Fixes: Committing Japanese IME text with Enter inserts an unintended newline in multiline
TextEditon Windows (#7876)Screencast: ✅ Japanese IME now behaves correctly while Korean IME behaves as before
This PR Fixes: Backspacing deletes characters during composition in certain Chinese IMEs (e.g., Sogou) on Windows (#7908)
Screencast: ✅ Sogou IME now behaves correctly
This PR Obsoletes #4794, because
eguireceives only IME events during composition from now onOn Windows, “incompatible” events are filtered in
egui-winit, aligning the behavior with other systems.Screencasts
Some Chinese IMEs on Windows:

The default Japanese IMEs on Windows:

The 2-set Korean IMEs handle arrow keys differently. It will be discussed in the next section.
This PR Reverts #4794, because it introduced several bugs
Some of its bugs have already been worked around in the past, but those workarounds might also be problematic. For example, #4912 is a workaround for a bug (#4908) introduced by #4794, and that workaround is in fact the root cause of the macOS backspacing bug I have worked around with #7810. (The reversion of #4912 is out of the scope of this PR, I will do that in #7983.)
It Caused: Arrow keys are incorrectly blocked during typical Korean IME composition
When composing Korean text using 2-Set IMEs, users should still be able to move the cursor with arrow keys regardless if the composition is committed.
Correct behavior
Screencasts
macOS TextEdit:

Windows Notepad:

With #4794 reverted,

eguialso behaves correctly (tested on Linux + Wayland, macOS, and Windows):Incorrect behavior caused by #4794
remove_ime_incompatible_eventsremoved arrow-key events in such cases. As a result, the first arrow key press only commits the composition, and users need to press the arrow key again to move the cursor:Screencast
This is essentially the same issue described here: #7877 (comment)
It Caused: Backspacing leaves the last character in Korean IME pre-edit text not removed on macOS
Screencasts
Before this PR:

After this PR:

Korean IMEs also use Enter to confirm Hanja selections, and will not work properly in the Korean “IME mode” proposed by #7898 (#7914)
Screencast: Korean IME using Enter and Space for confirmation (IBus Korean Hangul IME)
The screencast below demonstrates that some Korean IMEs handle Hanja selection in a way similar to Japanese IMEs: the Up/Down arrow keys are used to navigate candidates, and Enter confirms the selected candidate.
Screencasts: Another example
Using the built-in Korean IME on Windows, I type two lines: the first line in Hangul, and the second line as the same word converted to Hanja.
Correct behavior in Notepad (reference):

Behavior after applying this PR, which matches the Notepad behavior:

Behavior after applying #7914 with the “IME mode” set to Korean (which is also the behavior before this PR being applied):

On the second line, each time a Hanja character is confirmed, an unintended newline is inserted. This mirrors the Japanese IME issues that are supposed to be fixed by setting the “IME mode” to Japanese.
(These Japanese IME issues are fixed in this PR as mentioned before.)