Repository navigation
Use UTF-8 for text file I/O - #671
Conversation
Path.read_text(), Path.write_text() and open() without an encoding use the locale encoding, which on Windows is the ANSI code page (cp1252, cp949...) unless Python runs in UTF-8 mode. JupyterLab's schemas are UTF-8, so GET /lab/api/settings failed with UnicodeDecodeError on a Korean (cp949) Windows install, on the ellipsis in the "Connected Sessions…" label of @jupyterlab/running-extension. On cp1252 it does not fail but the label comes back as "Connected Sessions…". The contents API and nbconvert had the same problem when writing. Text files were written in the locale encoding but read back as UTF-8, so a saved "café" could not be opened again (404), and text the code page cannot encode failed to save and left the file empty. An HTML export of a notebook with such text failed with UnicodeEncodeError. Pass encoding="utf-8" to every text read and write in api/ and plugins/: the JSON files (schemas, package.json, overrides.json, page_config.json, language packs, kernelspecs, connection files), notebooks and text files in the contents API, the nbconvert output, the kernel environment file and the auth secret. The test runs the server in a new interpreter with -X warn_default_encoding and turns EncodingWarning from jupyverse code into an error, so it fails on every platform, not only on Windows. Fixes jupyter-server#557
|
The one failing job (Windows, Python 3.12) is a timeout in |
|
The failing |
davidbrochart
left a comment
There was a problem hiding this comment.
Thanks @Thanatos9404.
Fixes #557
Path.read_text(),Path.write_text()andopen()without anencodinguse the locale encoding, which on Windows is the ANSI code page unless Python runs in UTF-8 mode. JupyterLab's schemas are UTF-8, soGET /lab/api/settingsbreaks on a schema with a non-ASCII character. With JupyterLab 4.6.4 that is@jupyterlab/running-extension/plugin.json("Connected Sessions…"): the…starts at byte 943, and decoding that file as cp949 gives exactly the error in the issue. I don't have a Korean locale, but on Windows 11 with cp1252 the request goes through and the label comes back asConnected Sessions….Writes have the same problem, and there it loses data. On the same machine, through the REST API:
PUT /api/contents/cafe.txtwith"café"writesb'caf\xe9', and the nextGETreturns 404 ("Item not found"), becauseread_contentdecodes the file as UTF-8.日本語) returns 404 ("Error saving") and leaves the file empty, also when it had content before.GET /nbconvert/html/<notebook>for a notebook with日本語in a cell returns 500 (UnicodeEncodeError): the export is written in the locale encoding and then served as UTF-8.This passes
encoding="utf-8"to every text read and write inapi/andplugins/(25 calls in 8 files), which is what jupyterlab_server, jupyter_client and jupyter_server do for the same files. Binary reads and writes are unchanged. With the change, the label, the saved files and the HTML export come out right.Test
tests/test_encoding.pystarts jupyverse (withASGITransport) in a new interpreter with-X warn_default_encodingand turnsEncodingWarningfrom jupyverse modules into errors (and fromanyio, whosePathmethods open files in a worker thread), so it fails onmainon Linux and macOS too. It goes through the settings, kernelspecs, contents (with non-ASCII text), nbconvert and connection-file paths, which covers 14 of the 25 calls; the others are on paths it doesn't reach, such asoverrides.json, which is only read when it exists.Run on Windows 11 (cp1252):
pytest -v --timeout=60gives 179 passed on Python 3.10, 3.13 and 3.14 (178 onmain), andmypy ./api ./plugins,ruff checkandruff format --checkare clean.