Repository navigation
Save %%scalene cells as UTF-8 - #1106
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <[email protected]>
%%scalene and %scrun wrote the cell to _ipython-input-N-profile with the
locale encoding, but run_profiler() reads that file as UTF-8. On Windows
(cp1252, cp932) a cell with non-ASCII text either failed to write
(UnicodeEncodeError) or failed to load ("Scalene failed to initialize",
UnicodeDecodeError). Write the file as UTF-8, and do the same where the
tracer rewrites the cell file from the IPython history.
The Jupyter display server also read the generated HTML, which
generate_html() writes as UTF-8, with the locale encoding; read it as
UTF-8 too.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The narrowly scoped encoding changes align file readers and writers, with a focused regression test and no identified blocking issues.
No open issues 🎉
What changed in this PR
This PR fixes non-ASCII handling in Scalene’s Jupyter integration by explicitly using UTF-8 for cell files and generated HTML, matching their corresponding readers and writers.
Changes:
- Save magic cells and reconstructed IPython history cells as UTF-8.
- Read generated profile HTML as UTF-8 in the Jupyter display server.
- Add a subprocess regression test for non-ASCII cells under a non-UTF-8 locale.
| File | Description |
|---|---|
tests/test_magics_utf8.py |
Tests UTF-8 cell saving under a non-UTF-8 locale. |
scalene/scalene_tracing.py |
Writes reconstructed history cells as UTF-8. |
scalene/scalene_magics.py |
Saves cells with explicit UTF-8 encoding. |
scalene/scalene_jupyter.py |
Reads profile HTML with explicit UTF-8 encoding. |
Review effort: Balanced
This was referenced Oct 26, 2026
emeryberger
added a commit
that referenced
this pull request
Oct 26, 2026
ubuntu-latest now resolves to Ubuntu 26.04, for which actions/setup-python has no Python 3.9 build, so every 3.9 Ubuntu job failed at setup and fail-fast cancelled the rest of the matrix (e.g. #1106). Co-authored-by: Claude Opus 5.5 <[email protected]>
emeryberger
added a commit
that referenced
this pull request
Nov 13, 2026
ubuntu-latest now resolves to Ubuntu 26.04, for which actions/setup-python has no Python 3.9 build, so every 3.9 Ubuntu job failed at setup and fail-fast cancelled the rest of the matrix (e.g. #1106). Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Windows,
%%scalene/%scrunfails for a cell with non-ASCII text.ScaleneMagics.run_codewrites the cell to_ipython-input-N-profilewith the locale encoding, andrun_profilerreads it back as UTF-8 (utf-8-sigsince #1092). With cp932, a cell containingx = "日本語"gives:With cp1252,
"café"fails the same way, and a character the code page can't encode fails while writing (UnicodeEncodeError).This writes the cell file as UTF-8. Two other spots use the locale encoding for files that are UTF-8:
_handle_jupyter_cellinscalene_tracing.pyrewrites the same_ipython-input-*file from the IPython history, andscalene_jsonreads it as UTF-8. I didn't reach this path from a terminal IPython session, but it writes the same file.scalene_jupyter.pyread the HTML thatgenerate_htmlwrites as UTF-8; the few non-ASCII characters in the embedded bundle (e.g. the currency symbols in the tablesort number regex) came out garbled.tests/test_magics_utf8.pyruns the magic in a child process with a non-UTF-8 locale (LC_ALL=C,PYTHONUTF8=0,PYTHONCOERCECLOCALE=0: ASCII on Linux/macOS, the ANSI code page on Windows) with a mocked IPython history andrun_profiler, and checks that the saved file decodes as UTF-8 to the cell. It fails without the fix (UnicodeEncodeError: 'cp932' codec can't encode character '\xe9') and passes with it.I used Claude Code to write this change.