Repository navigation
Read voices.json and config.json as UTF-8 so KittenTTS 2 loads on Windows - #164
Conversation
Fixes the UnicodeDecodeError in KittenML#158. Signed-off-by: kevin9327 <[email protected]>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
rahul05ranjan
left a comment
rahul05ranjan
left a comment
There was a problem hiding this comment.
Verified the technique end to end on Linux: with PYTHONUTF8=0, PYTHONCOERCECLOCALE=0, LC_ALL=C the child interpreter's preferred encoding is ANSI_X3.4-1968, and a bare open() on a UTF-8 JSON fixture with non-ASCII text dies with UnicodeDecodeError at the first non-ASCII byte — the same failure class as the cp1252 report in #158. So the tests reproduce the Windows problem for the right reason and encoding="utf-8" is the correct fix.
Also checked the "only text-mode open() calls left in the package" claim: the remaining open( occurrences are safe_open() (safetensors binary handles) in kittenml/kittentts2/weights.py:35 and kittenml/kittentts2/tl2_emb4.py:146, so nothing text-mode is left bare.
Two details in the test design that I appreciate after hitting the second one myself: the guard test asserting the child really isn't UTF-8 (without it the other tests would pass vacuously), and having the child print ascii(...) — an ASCII stdout can't fail the test on its own output, which my first reproduction of this setup did hit when printing raw non-ASCII.
Nothing to change from my side. The only adjacent case not covered is a BOM-prefixed voices.json (which encoding="utf-8" + json.load still rejects), but that file ships from the model repo and has no BOM, so utf-8-sig isn't needed here.
Fixes the first half of #158 (the
UnicodeDecodeError).Symptom: on Windows, loading
KittenML/kitten-tts-2fails at startup in_load_voiceswithUnicodeDecodeError: 'charmap' codec can't decode byte 0x9d. The shippedvoices.jsonis UTF-8 and has non-ASCII characters in it, butopen()without an encoding uses the locale code page (cp1252 on Windows).Fix: pass
encoding="utf-8"inKittenTTS2._load_voicesand in bothopen()calls inkittenml/repo.resolve_repo. The second one readsconfig.jsonand has the same problem. These were the only text-modeopen()calls left in the package.Tests:
tests/test_json_encoding.pyruns each loader in a child interpreter whose default encoding is ASCII (PYTHONUTF8=0,LC_ALL=C,PYTHONCOERCECLOCALE=0). That stands in for cp1252 on Linux and macOS CI. A guard test checks that the child really isn't using UTF-8. Both loader tests fail onmainwithUnicodeDecodeErrorand pass with this change.python -m unittest tests.test_json_encoding tests.test_text_normalizationgives 9 passed.This PR doesn't touch the
torch.acceleratorimport crash from the second half of #158. That one comes from transformers and diffusers, not from this repo.I prepared this change with AI assistance and reviewed it and the tests myself.