Skip to content

Use UTF-8 for text file I/O - #671

Merged
davidbrochart merged 1 commit into
jupyter-server:mainfrom
Thanatos9404:utf8-file-io
Oct 19, 2026
Merged

davidbrochart merged 1 commit into
jupyter-server:mainfrom
Thanatos9404:utf8-file-io

Conversation

@Thanatos9404

Copy link
Copy Markdown
Contributor

Fixes #557

Path.read_text(), Path.write_text() and open() without an encoding use the locale encoding, which on Windows is the ANSI code page unless Python runs in UTF-8 mode. JupyterLab's schemas are UTF-8, so GET /lab/api/settings breaks on a schema with a non-ASCII character. With JupyterLab 4.6.4 that is @jupyterlab/running-extension/plugin.json ("Connected Sessions…"): the … starts at byte 943, and decoding that file as cp949 gives exactly the error in the issue. I don't have a Korean locale, but on Windows 11 with cp1252 the request goes through and the label comes back as Connected Sessions….

Writes have the same problem, and there it loses data. On the same machine, through the REST API:

  • PUT /api/contents/cafe.txt with "café" writes b'caf\xe9', and the next GET returns 404 ("Item not found"), because read_content decodes the file as UTF-8.
  • Saving text that cp1252 can't encode (日本語) returns 404 ("Error saving") and leaves the file empty, also when it had content before.
  • GET /nbconvert/html/<notebook> for a notebook with 日本語 in a cell returns 500 (UnicodeEncodeError): the export is written in the locale encoding and then served as UTF-8.

This passes encoding="utf-8" to every text read and write in api/ and plugins/ (25 calls in 8 files), which is what jupyterlab_server, jupyter_client and jupyter_server do for the same files. Binary reads and writes are unchanged. With the change, the label, the saved files and the HTML export come out right.

Test

tests/test_encoding.py starts jupyverse (with ASGITransport) in a new interpreter with -X warn_default_encoding and turns EncodingWarning from jupyverse modules into errors (and from anyio, whose Path methods open files in a worker thread), so it fails on main on Linux and macOS too. It goes through the settings, kernelspecs, contents (with non-ASCII text), nbconvert and connection-file paths, which covers 14 of the 25 calls; the others are on paths it doesn't reach, such as overrides.json, which is only read when it exists.

Run on Windows 11 (cp1252): pytest -v --timeout=60 gives 179 passed on Python 3.10, 3.13 and 3.14 (178 on main), and mypy ./api ./plugins, ruff check and ruff format --check are clean.

Path.read_text(), Path.write_text() and open() without an encoding use
the locale encoding, which on Windows is the ANSI code page (cp1252,
cp949...) unless Python runs in UTF-8 mode. JupyterLab's schemas are
UTF-8, so GET /lab/api/settings failed with UnicodeDecodeError on a
Korean (cp949) Windows install, on the ellipsis in the
"Connected Sessions…" label of @jupyterlab/running-extension. On cp1252
it does not fail but the label comes back as "Connected Sessions…".

The contents API and nbconvert had the same problem when writing. Text
files were written in the locale encoding but read back as UTF-8, so a
saved "café" could not be opened again (404), and text the code page
cannot encode failed to save and left the file empty. An HTML export of
a notebook with such text failed with UnicodeEncodeError.

Pass encoding="utf-8" to every text read and write in api/ and plugins/:
the JSON files (schemas, package.json, overrides.json, page_config.json,
language packs, kernelspecs, connection files), notebooks and text files
in the contents API, the nbconvert output, the kernel environment file
and the auth secret.

The test runs the server in a new interpreter with
-X warn_default_encoding and turns EncodingWarning from jupyverse code
into an error, so it fails on every platform, not only on Windows.

Fixes jupyter-server#557
@Thanatos9404

Copy link
Copy Markdown
Contributor Author

The one failing job (Windows, Python 3.12) is a timeout in plugins/ystore_sqlite/tests/test_db.py::test_move_db[asyncio], which this PR doesn't touch. The same test timed out on main in https://github.com/jupyter-server/jupyverse/actions/runs/36481289805, and the other four Windows jobs pass here, so I think it's the existing flake. Could you re-run it when you have a moment? I can't re-run jobs from a fork.

@Thanatos9404

Copy link
Copy Markdown
Contributor Author

The failing Run tests (windows, 3.12) job is a timeout in plugins/ystore_sqlite/tests/test_db.py::test_move_db, which this PR does not touch. The same timeout happened on #665 (a pre-commit autoupdate), so it looks flaky on Windows. Could you re-run it when you have a moment?

@davidbrochart davidbrochart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Thanatos9404.

@davidbrochart
davidbrochart merged commit b6712de into jupyter-server:main Oct 19, 2026
32 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Open JupyterLab through Jupyverse got UnicodeDecodeError

2 participants