Repository navigation
ipynb: make ANSI escape stripping linear. - #12007
Merged
Merged
Conversation
filterIpynbOutput stripped ANSI escapes from output code blocks by rebuilding the text one character at a time with T.cons, so time was quadratic in the length of the output. A notebook with one 32,000-line stream output took 4.3 s to convert to HTML; the notebooks attached to jgm#7744 took 42 s. Split at each ESC [ with T.breakOn and join the pieces once instead. Output is unchanged. Add test/command/7744.md, since nothing tested the stripping itself. Closes jgm#7744.
JayOfTheKeyboard
added a commit
to JayOfTheKeyboard/JayOfTheKeyboard
that referenced
this pull request
Oct 28, 2026
Owner
|
thanks! |
JayOfTheKeyboard
added a commit
to JayOfTheKeyboard/JayOfTheKeyboard
that referenced
this pull request
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7744.
When the input is ipynb and the output format is not,
filterIpynbOutputrunsremoveANSIEscapesover every code block in the chosen outputs. It rebuilt the text one character at a time withT.cons, which copies the rest of the text each time, so the time grows with the square of the output's length.--ipynb-output=allskips the filter, which is why it was always fast.The notebooks in #7744 are slow for the same reason. Their
+jsondata becomes a code block in the output and goes through this function; the escaped quotes matter only because they make the string long. Timings with-s -t html, on currentmain(60725b8, GHC 9.12.4) and with this change:holoview-load-only-manual_filtered.ipynbfrom #7744holoview-load-only.ipynbfrom #7744(The 3.13 release binary is slower still: 7.9 s for 4,000 lines, 31.6 s for 8,000, and it did not finish the #7744 notebooks within two minutes.)
The function now splits the text at each
ESC [withT.breakOn, drops everything up to and including the nextm, as before, and joins the pieces with oneT.concat. The output is unchanged. I compared the old and new functions on every string of up to 8 characters drawn fromESC,[,m,3,aandé(about 2 million strings), on 200,000 random strings of up to 400 characters that also include CJK, an emoji and newlines, and on cases such as a loneESC,ESC (Band an unterminatedESC [31at the end. None differed. The HTML output for all the inputs above is byte-for-byte the same, and so is the Markdown output for 201 real notebooks, 29 of them with ANSI escapes.The escape handling is otherwise the same. It still removes only sequences that end in
m, and anESC [that is not closed still removes everything up to the nextmanywhere later in the text, or to the end if there is none. So every CSI sequence is treated as running to the nextm, and cursor and erase codes such asESC [2KorESC [K, which tqdm and pip progress bars write, delete real text:ESC [2Kfollowed by a newline andloading modelcomes out asodel. I kept that behaviour on purpose, since fixing it changes the output; it could be a follow-up. The same goes for dropping+jsondata from non-ipynb output, as discussed in #7744.Test:
test/command/7744.md. Nothing tested the escape stripping before, and the test pins its output, not its speed. It covers SGR sequences next to non-ASCII text,[0mwithout anESCandESC (B(both kept), back-to-back sequences, anESC [2Kwhose removal runs on to themin the next line, sequences interrupted before theirmby anotherESC [or byESC (B, and an unterminated sequence at the end of an output. It passes before and after the change, since the old code gave the right output, just slowly; a timing test didn't seem to fit the test suite. Breaking the new code in each of six ways (keeping them, matching a bareESC, dropping the last piece, stopping at the wrong character, ending a sequence at a newline, restarting at an innerESC) makes the test fail.The full test suite passes (4492 tests, GHC 9.12.4,
-Werror).Prepared with Claude Code.