@katainaka0503, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bpasero and @egamma to be potential reviewers.

@katainaka0503,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@katainaka0503
could be better used files.encoding = 'auto' instead of files.autoDetectEncoding ?

@buzzzzer
I had considered that before I implemented this. And I realized that when opening a new file, VSCode can't specify preferred encoding if 'files.encoding' is set to 'auto'.

Thanks for this PR, current status is that we are reviewing the implications of JSChardets LGPL license for VS Code.

Rebased.

I expect this feature to be merged into VSCode. Thanks very much.

No status update yet from what I wrote here.

Any progress? Thanks.

I think it is no use rushing.

Rebased

@katainaka0503,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@katainaka0503 have you thought about allowing a user to ad-hoc detect the encoding of an opened file from the encoding picker?

From a file, click on the encoding and pick the option to Reopen with Encoding:

In this list, the first entry (maybe with a separator) could be to automatically detect the encoding from the file:

One thing to keep in mind is that currently we only allow the first 512 bytes of the file to be used when detecting the encoding. This might cause some encodings to not get detected properly. If we have an explicit action for detection, we could maybe increase the buffer size to give the detection a higher chance of finding an encoding.

Otherwise this PR seems well programmed, good job. Still waiting on the license approval.

@bpasero Thanks! I am very pleased.

I think it's very good if it is able to select automatically detect encoding from encoding picker.
But I don't like to select the entry every time I open the file.

I think one of the problems is the word detect implies vscode can detect the encoding properly every time, so maybe the word guess should be used instead for example.

And also In the description of files.autoDetectEncoding , it's should be written that Files may be detected inappropriately. for example.

What's do you think?

@katainaka0503 yes, this would be an optional thing to "guess" the encoding from the picker in case the user does not want to opt in to the setting for all files. Since we already have jschardet in the system, this functionality could be added quite easily I guess.

As for the setting, maybe it would be better to rename it to autoGuessEncoding and have the wording be When enabled, will attempt to guess the character set encoding when opening files.

I would then also suggest to rename the flag from autoDetectEncoding to autoGuessEncoding.

As you probably already figured out, we are using iconv-lite to convert to and from encodings. If jschardet is returning an encoding with a specific name, we have to ensure that this is the same name used in iconv-lite (see here for a list of supported encodings in iconv-lite). If the names are not matching, the check for encodingExists would fail.

Finally I am running into some issues when testing this and I think this exposes a problem with encoding auto detection in general. After making changes to a file and reopening it, the detected encoding is suddenly different and characters cannot be displayed anymore. Steps to reproduce:

• create a file with the contents below saved as cp1250 encoding (I was using Sublime Text for this)
• open the file in VS Code with encoding detection, it gets opened as ISO-8859-2
• add the characters öäü to the end of the file in a newline
• reopen the file and notice how the encoding is now detected as windows-1255

Contents of file:

// A kódolás szinte minden adatátviteli és kommunikációs rendszerben használható, és a következõ európai nyelvek megjelenítésére alkalmas: bosnyák, cseh, horvát, lengyel, magyar, román, szerb (a latinbetÿs írásmóddal), szerbhorvát, szlovák, szlovén, alsó-szorb és felsõ-szorb.

After reopening:

@bpasero Ok, I understand.
And I'll implement this.

Then, the remaining tasks are

• Fix description.

• Rename autoDetectEncoding to autoGuessEncoding.

• Research whether jschardet can be configured to return only with higher confidence

• Research difference of encoding names between jschardet and iconv-lite

• Add autoGuessEncoding entry in encoding picker.

Thanks for precise test!

@bpasero I found there are two ways to add entry in encoding picker.

One is to add autoGuessEncoding to encoding picker.
The another is to add Shift_JIS(Guessed from content) or ISO-8859-2(Guessed from content).

I think the latter is better. Because users can decide based on result of guessing encoding.

Maybe we go with what we have now and wait for more user information on how this behaves in the specific cases.

I added a guessed encoding entry in encoding picker as a prototype.
Now, using buffer size 512 byte.

Merged. did some slight changes to encoding picker to prevent showing encodings multiple times.

@bpasero Thanks!

@katainaka0503 just so that you know, there is a number of cases already where encoding guessing does not work: aadsm/jschardet#31, aadsm/jschardet#29, aadsm/jschardet#30

@bpasero Thanks for letting me know! I'll check them.