fix: switch CharacterMode from RGui to LinkDLL to prevent system() hang on Windows#117
Merged
fix: switch CharacterMode from RGui to LinkDLL to prevent system() hang on Windows#117
system() hang on Windows#117Conversation
…ng on Windows R's do_system() invalidates standard handles (SetStdHandle with INVALID_HANDLE_VALUE) when CharacterMode is RGui, causing system() and system2() calls to hang indefinitely on Windows. Fix by switching CharacterMode to LinkDLL after R_SetParams() but before setup_Rmainloop(), following the same pattern used by sircon. This keeps the console callback setup from RGui mode while avoiding the handle invalidation in do_system(). Closes: #116 Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Add compile-time assert that UImode (repr(u32)) matches c_int size - Upgrade log level from warn to error when CharacterMode symbol cannot be loaded, since system() will hang without it Co-Authored-By: Claude Opus 4.6 <[email protected]>
libloading's Windows get::<T>() requires size_of::<T>() == size_of::<FARPROC>() (8 bytes on 64-bit). Using c_int (4 bytes) caused IncompatibleSize error, silently returning null pointers for all global int variables (R_Interactive, R_SignalHandlers, R_running_as_main_program, CharacterMode). This was the root cause of CharacterMode failing to load, which meant the RGui-to-LinkDLL switch never happened and system() still hung. Fix by using get::<usize>() with try_as_raw_ptr() to obtain the symbol address without the size constraint. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The try_as_raw_ptr() calls are already inside an outer unsafe block, so inner unsafe blocks are redundant and trigger warnings. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
eitsupi
changed the title
system() hang on Windows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CharacterModefromRGuitoLinkDLLafterR_SetParams()to fixsystem()/system2()hanging indefinitely on Windowsdo_system()invalidates standard handles viaSetStdHandle(INVALID_HANDLE_VALUE)whenCharacterModeisRGui, causing child processes to hanglibloadingsymbol loading for Windows globalintvariables —get::<c_int>()(4 bytes) silently failed thesize_of::<T>() == size_of::<FARPROC>()check (8 bytes on 64-bit), returning null pointers forCharacterMode,R_Interactive,R_SignalHandlers, andR_running_as_main_programTest plan
system("echo hello")and confirm it completes without hangingRcpp::cppFunction("int add(int x, int y) { return x + y; }")and confirm compilation succeeds:help print), help display, andmenu()still work correctlyCloses #116