Skip to content

ipynb: make ANSI escape stripping linear. - #12007

Merged
jgm merged 1 commit into
jgm:mainfrom
JayOfTheKeyboard:ipynb-ansi-linear
Oct 28, 2026
Merged

jgm merged 1 commit into
jgm:mainfrom
JayOfTheKeyboard:ipynb-ansi-linear

Conversation

@JayOfTheKeyboard

Copy link
Copy Markdown
Contributor

Closes #7744.

When the input is ipynb and the output format is not, filterIpynbOutput runs removeANSIEscapes over every code block in the chosen outputs. It rebuilt the text one character at a time with T.cons, which copies the rest of the text each time, so the time grows with the square of the output's length. --ipynb-output=all skips the filter, which is why it was always fast.

The notebooks in #7744 are slow for the same reason. Their +json data becomes a code block in the output and goes through this function; the escaped quotes matter only because they make the string long. Timings with -s -t html, on current main (60725b8, GHC 9.12.4) and with this change:

input main this PR
stream output, 8,000 lines 0.35 s 0.09 s
stream output, 16,000 lines 1.11 s 0.13 s
stream output, 32,000 lines 4.31 s 0.21 s
holoview-load-only-manual_filtered.ipynb from #7744 42.3 s 1.3 s
holoview-load-only.ipynb from #7744 41.7 s 1.4 s

(The 3.13 release binary is slower still: 7.9 s for 4,000 lines, 31.6 s for 8,000, and it did not finish the #7744 notebooks within two minutes.)

The function now splits the text at each ESC [ with T.breakOn, drops everything up to and including the next m, as before, and joins the pieces with one T.concat. The output is unchanged. I compared the old and new functions on every string of up to 8 characters drawn from ESC, [, m, 3, a and é (about 2 million strings), on 200,000 random strings of up to 400 characters that also include CJK, an emoji and newlines, and on cases such as a lone ESC, ESC (B and an unterminated ESC [31 at the end. None differed. The HTML output for all the inputs above is byte-for-byte the same, and so is the Markdown output for 201 real notebooks, 29 of them with ANSI escapes.

The escape handling is otherwise the same. It still removes only sequences that end in m, and an ESC [ that is not closed still removes everything up to the next m anywhere later in the text, or to the end if there is none. So every CSI sequence is treated as running to the next m, and cursor and erase codes such as ESC [2K or ESC [K, which tqdm and pip progress bars write, delete real text: ESC [2K followed by a newline and loading model comes out as odel. I kept that behaviour on purpose, since fixing it changes the output; it could be a follow-up. The same goes for dropping +json data from non-ipynb output, as discussed in #7744.

Test: test/command/7744.md. Nothing tested the escape stripping before, and the test pins its output, not its speed. It covers SGR sequences next to non-ASCII text, [0m without an ESC and ESC (B (both kept), back-to-back sequences, an ESC [2K whose removal runs on to the m in the next line, sequences interrupted before their m by another ESC [ or by ESC (B, and an unterminated sequence at the end of an output. It passes before and after the change, since the old code gave the right output, just slowly; a timing test didn't seem to fit the test suite. Breaking the new code in each of six ways (keeping the m, matching a bare ESC, dropping the last piece, stopping at the wrong character, ending a sequence at a newline, restarting at an inner ESC) makes the test fail.

The full test suite passes (4492 tests, GHC 9.12.4, -Werror).

Prepared with Claude Code.

filterIpynbOutput stripped ANSI escapes from output code blocks by
rebuilding the text one character at a time with T.cons, so time was
quadratic in the length of the output. A notebook with one 32,000-line
stream output took 4.3 s to convert to HTML; the notebooks attached to
jgm#7744 took 42 s. Split at each ESC [ with T.breakOn and join the pieces
once instead. Output is unchanged.

Add test/command/7744.md, since nothing tested the stripping itself.

Closes jgm#7744.
JayOfTheKeyboard added a commit to JayOfTheKeyboard/JayOfTheKeyboard that referenced this pull request Oct 28, 2026
@jgm
jgm merged commit 8605c4a into jgm:main Oct 28, 2026
9 of 10 checks passed
@jgm

jgm commented Oct 28, 2026

Copy link
Copy Markdown
Owner

thanks!

JayOfTheKeyboard added a commit to JayOfTheKeyboard/JayOfTheKeyboard that referenced this pull request Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ipynb reader performance: JSON escaped double quote quadratic in time

2 participants