To reproduce:

import time
for i in range(10):
    print ("\r" + "#" * i),
    time.sleep(0.1)
print "Step 2"
for i in range(10):
    print ("\r" + "#" * i),
    time.sleep(0.1)

should produce:

#########
Step 2
#########

Thanks for bringing this up; we have the same problem on aleph.sagemath.org with our use of the IPython messaging protocol.

I had a look at the save/restore, and it's a bit problematic.

When we write the ipynb to JSON, we use splitlines() for the text. So roundtrip to files, '\r#' becomes \n#. The change needed for this is to use splitlines(True), and ''.join(lines) instead of splitlines() and '\n'.join(lines). I believe I have a safe fix that will behave properly for notebooks written before/after the fix, but I will have to test carefully to make sure it doesn't cause any problems.

I do want to make a point of discouraging actually using terminal-centric behaviors in the notebook (\r, and ANSI colors, etc.). It's important that people don't think the notebook is a terminal emulator, and it never will be, nor should it be. But \r is simple enough.

After PR #1663, this should load from saves properly without further changes.

I think I've corrected the test in which to not add a div when the content is empty. This merged with #1663 also fixes the saving/loading issue.

Thanks for helping with this!

thanks, merged.

Crap - I merged this prematurely, and it caused a pretty critical failure (try !ls, and you will see no output at all). I reverted the original patch, so can you submit a new PR after testing the output of !ls?

Hang on, I've just understood this issue. So if there's a percentage counter that gets overwritten 100 times, we store every percentage, separated by \r characters? That seems unnecessary. Shouldn't we handle \r for display, but then just save the final state, as the user sees it when they hit save?

Correct, the notebook structure is a transcript of what actually happened, excluding any UI-specific interpretation. \r is a frontend-specific special character, which is interpreted (in this PR) in the notebook as clearing the preceding line, which is distinct from the terminal, where it simply moves the cursor, and discards no information until overwrites start happening.

This is a difference between \r and clear_output, and further reason why real notebook operations are better than faked terminal-behavior in the notebook. This is acceptable to me, because I consider \r support like this decidedly second-class, and have no problem with it suffering such a disadvantage.

We could store the transformed output, but we would have to be careful that we do not store the escaped HTML output (we had this bug before). If we make that change, then we must do fixCR always on the raw text and before fixConsole, because we would be saving the results of fixCR, and must never save the results of fixConsole.

I'd side with removing overwritten content at some point before saving. Even if we'd rather they didn't, people are going to use \r for progress counters, because it works and library authors shouldn't have to detect and handle the notebook. If it's left in, it's extra noise for version control. Also, if one day another application can load ipynb files, but can't handle \r, its users won't appreciate seeing a load of messy progress information.

Do you want to add that as a criterion for PR #1674 (note that this one is already closed)?

I think it does make sense to do, my main point is that I don't think it's worth the effort. If someone does feel like doing it, they are welcome, but must be very careful with the points I mentioned.