Skip to content

Windows: Change command line interface to UTF-8#12377

Open
larskanis wants to merge 1 commit intoruby:masterfrom
larskanis:utf8-cli
Open

Windows: Change command line interface to UTF-8#12377
larskanis wants to merge 1 commit intoruby:masterfrom
larskanis:utf8-cli

Conversation

@larskanis
Copy link
Copy Markdown
Contributor

@larskanis larskanis commented Dec 17, 2024 •
edited
Loading

The following inputs are changed to consistently work with UTF-8:

  • script name
  • include paths
  • script input from stdin

They are currently a inconsistent mix of locale encoding and UTF-8, depending on the situation.

Given the following script and a codepage of 850 it changes from: to to:

# execute with: ruby -It€st täst-locale-enc.rb
def pr(*strs)
  strs.each do |str|
    p [str, IO===str ? str.external_encoding&.name : str.encoding.name]
  end
end

if $0==__FILE__
  pr STDIN      # from: [#<IO:<STDIN>>, "CP850"]
                # to:   [#<IO:<STDIN>>, "UTF-8"]
  pr $0         # from: ["t\x84st-locale-enc.rb", "CP850"]
                # to:   ["täst-locale-enc.rb", "UTF-8"]
  pr __FILE__   # from: ["t\x84st-locale-enc.rb", "CP850"]
                # to:   ["täst-locale-enc.rb", "UTF-8"]
  pr __dir__    # from: ["C:/Users/Lars/ruby", "CP850"]
                # to:   ["C:/Users/Lars/ruby", "UTF-8"]
  pr 'ä'        # from: ["ä", "UTF-8"]
                # to:   ["ä", "UTF-8"]
  pr '€'        # from: ["€", "UTF-8"]
                # to:   ["€", "UTF-8"]
  pr $:.first   # from: ["C:/Users/Lars/ruby/t\xE2\x82\xACst", "ASCII-8BIT"]
                # to:   ["C:/Users/Lars/ruby/t€st", "UTF-8"]
  pr $:.last    # from: ["c:/Ruby34/lib/ruby/3.4.0+1/aarch64-mingw-ucrt", "CP850"]
                # to:   ["C:/Users/Lars/ruby/lib/ruby/3.4.0+1/aarch64-mingw-ucrt", "UTF-8"]
end

... and with code from STDIN the changes look like so:

# execute with: cat täst-locale-enc.rb | ruby -It€st
if $0==__FILE__
  pr STDIN      # from: [#<IO:<STDIN>>, "UTF-8"]
                # to:   [#<IO:<STDIN>>, "UTF-8"]
  pr $0         # from: ["-", "CP850"]
                # to:   ["-", "UTF-8"]
  pr __FILE__   # from: ["-", "UTF-8"]
                # to:   ["-", "UTF-8"]
  pr __dir__    # from: [".", "US-ASCII"]
                # to:   [".", "US-ASCII"]
  pr 'ä'        # from: ["\xC3\xA4", "CP850"]
                # to:   ["ä", "UTF-8"]
  pr '€'        # from: ["\xE2\x82\xAC", "CP850"]
                # to:   ["€", "UTF-8"]
  pr $:.first   # from: ["C:/Users/Lars/ruby/t\xE2\x82\xACst", "ASCII-8BIT"]
                # to:   ["C:/Users/Lars/ruby/t€st", "UTF-8"]
  pr $:.last    # from: ["c:/Ruby34/lib/ruby/3.4.0+1/aarch64-mingw-ucrt", "CP850"]
                # to:   ["C:/Users/Lars/ruby/lib/ruby/3.4.0+1/aarch64-mingw-ucrt", "UTF-8"]
end

This PR doesn't change STDIN encoding when keystrokes are read from the console. They are still in locale encoding. This is because it is difficult to implement UTF-8 there and it is a different topic than the command line interface.

This PR works equally with Prism and Parse.y parser.

Closes [Bug #20774]
Closes [Bug #20699]
Closes oneclick/rubyinstaller2#265

@larskanis
Copy link
Copy Markdown
Contributor Author

larskanis commented Dec 18, 2024

The test is modified to show, that arbitrary Unicode characters can work now. However it is not necessary to change the test - it passes with and without modification.

@larskanis
Copy link
Copy Markdown
Contributor Author

larskanis commented Dec 18, 2024

It would be nice if this PR could be merged to Ruby-3.4 since it fixes longstanding bugs. I added an item to the compatibility section of the NEWS.md.

@launchable-app

This comment has been minimized.

Sign in to view
@larskanis
Copy link
Copy Markdown
Contributor Author

larskanis commented May 27, 2025 •
edited
Loading

I rebased this and hope this can be merged to ruby-4.0.

@larskanis larskanis force-pushed the utf8-cli branch 2 times, most recently from a9eb74f to 84571e0 Compare December 17, 2025 12:09
larskanis added a commit to oneclick/rubyinstaller2-packages that referenced this pull request Dec 20, 2025
@larskanis
Add patch for UTF-8 CLI from ruby-core PR 12377
ddda949 
ruby/ruby#12377

This is proposed and asked since one year, but Ruby core team seems too busy to care about it.
@larskanis larskanis force-pushed the utf8-cli branch 3 times, most recently from 74d3baa to 38331ec Compare December 22, 2025 18:45
@larskanis
Copy link
Copy Markdown
Contributor Author

larskanis commented Dec 23, 2025

I added this patch to RubyInstaller, so that it's part of rubyinstaller-head + github CI since 2025-12-20 : oneclick/rubyinstaller2-packages@ddda949 . If there are no objections, it will be part of RubyInstaller-4.0.

@larskanis
Windows: Change command line interface to UTF-8
f542f7a 
... even when the Windows codepage isn't set to 65001/UTF-8.
A non-UTF-8 codepage is still the default on Windows-11.

The following inputs are changed to consistently work with UTF-8:
- script name
- include paths
- script input from stdin

They are currently a inconsistent mix of locale encoding and UTF-8, depending on the situation.

The adjusted tests don't trigger encoding issues when the codepage is 65001 (UTF-8), but they do on 850 or 437 or so.
Therefore it doesn't fail on Github Actions, which sets the codepage to UTF-8 per default.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ruby require fails when the path has special characters

1 participant

@larskanis