Skip to content

Save %%scalene cells as UTF-8 - #1106

Merged
emeryberger merged 3 commits into
plasma-umass:masterfrom
b84330808:jupyter-cell-utf8
Oct 26, 2026
Merged

emeryberger merged 3 commits into
plasma-umass:masterfrom
b84330808:jupyter-cell-utf8

Conversation

@b84330808

Copy link
Copy Markdown
Contributor

On Windows, %%scalene / %scrun fails for a cell with non-ASCII text. ScaleneMagics.run_code writes the cell to _ipython-input-N-profile with the locale encoding, and run_profiler reads it back as UTF-8 (utf-8-sig since #1092). With cp932, a cell containing x = "日本語" gives:

Scalene failed to initialize.
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x93 in position 7: invalid start byte

With cp1252, "café" fails the same way, and a character the code page can't encode fails while writing (UnicodeEncodeError).

This writes the cell file as UTF-8. Two other spots use the locale encoding for files that are UTF-8:

  • _handle_jupyter_cell in scalene_tracing.py rewrites the same _ipython-input-* file from the IPython history, and scalene_json reads it as UTF-8. I didn't reach this path from a terminal IPython session, but it writes the same file.
  • The Jupyter display server in scalene_jupyter.py read the HTML that generate_html writes as UTF-8; the few non-ASCII characters in the embedded bundle (e.g. the currency symbols in the tablesort number regex) came out garbled.

tests/test_magics_utf8.py runs the magic in a child process with a non-UTF-8 locale (LC_ALL=C, PYTHONUTF8=0, PYTHONCOERCECLOCALE=0: ASCII on Linux/macOS, the ANSI code page on Windows) with a mocked IPython history and run_profiler, and checks that the saved file decodes as UTF-8 to the cell. It fails without the fix (UnicodeEncodeError: 'cp932' codec can't encode character '\xe9') and passes with it.

I used Claude Code to write this change.

b84330808 and others added 2 commits October 26, 2026 17:08
%%scalene and %scrun wrote the cell to _ipython-input-N-profile with the
locale encoding, but run_profiler() reads that file as UTF-8. On Windows
(cp1252, cp932) a cell with non-ASCII text either failed to write
(UnicodeEncodeError) or failed to load ("Scalene failed to initialize",
UnicodeDecodeError). Write the file as UTF-8, and do the same where the
tracer rewrites the cell file from the IPython history.

The Jupyter display server also read the generated HTML, which
generate_html() writes as UTF-8, with the locale encoding; read it as
UTF-8 too.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The narrowly scoped encoding changes align file readers and writers, with a focused regression test and no identified blocking issues.

No open issues 🎉

What changed in this PR

This PR fixes non-ASCII handling in Scalene’s Jupyter integration by explicitly using UTF-8 for cell files and generated HTML, matching their corresponding readers and writers.

Changes:

  • Save magic cells and reconstructed IPython history cells as UTF-8.
  • Read generated profile HTML as UTF-8 in the Jupyter display server.
  • Add a subprocess regression test for non-ASCII cells under a non-UTF-8 locale.
File Description
tests/​test_magics_utf8.py Tests UTF-8 cell saving under a non-UTF-8 locale.
scalene/​scalene_tracing.py Writes reconstructed history cells as UTF-8.
scalene/​scalene_magics.py Saves cells with explicit UTF-8 encoding.
scalene/​scalene_jupyter.py Reads profile HTML with explicit UTF-8 encoding.

Review effort: Balanced

emeryberger added a commit that referenced this pull request Oct 26, 2026
ubuntu-latest now resolves to Ubuntu 26.04, for which actions/setup-python
has no Python 3.9 build, so every 3.9 Ubuntu job failed at setup and
fail-fast cancelled the rest of the matrix (e.g. #1106).

Co-authored-by: Claude Opus 5.5 <[email protected]>
@emeryberger
emeryberger merged commit 9643e18 into plasma-umass:master Oct 26, 2026
41 checks passed
emeryberger added a commit that referenced this pull request Nov 13, 2026
ubuntu-latest now resolves to Ubuntu 26.04, for which actions/setup-python
has no Python 3.9 build, so every 3.9 Ubuntu job failed at setup and
fail-fast cancelled the rest of the matrix (e.g. #1106).

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants