Repository navigation
fix: keep the AI analysis from crashing on a non-UTF-8 stdout - #3096
Merged
Merged
Conversation
Third of the same shape as soxoj#3050 (startup banners) and soxoj#3056 (result lines and the spinner). The one terminal write in the AI module that was not covered is the analysis itself. _stream_response() writes each token of the model's reply straight to stdout. The reply is arbitrary text, and an OSINT run on an international username routinely comes back with Cyrillic or CJK. On Windows, Python takes stdout's encoding from the process ANSI codepage -- cp1252 on a default install -- so that write raises UnicodeEncodeError, the exception leaves the coroutine, and the analysis ends mid-sentence with the tokens already printed left on screen. sys.stdout's error handler is surrogateescape, which is not a guard here: it rescues lone surrogates only, so an ordinary unencodable character still raises. There is a test that pins exactly that, so this suite notices if the platform ever changes. Also corrects the _frames_for docstring, which I wrote in soxoj#3056. It says the braille frames "raised UnicodeEncodeError inside the spinner's daemon thread". They did not: unlike stdout, stderr's error handler is backslashreplace, so the write never raised -- it printed the escape, and the animation became a column of literal ⠋. The fix was right, the reason given for it was not, and the difference between the two streams is exactly what this change turns on. print_streaming() writes to stdout the same way and has the same defect, but it has no callers anywhere in the tree, so it is left alone rather than changed without a way to exercise it. Verified: 6 tests added to tests/test_ai.py, driven by real cp1252 streams rather than mocks so the codec decides. Three mutations, all killed. Full suite on Windows: the set of failing test names is identical before and after (35 pre-existing).
ntdatt812
force-pushed
the
fix/ai-output-encoding
branch
from
September 7, 2026 09:06
5d2d7b0 to
613c419
Compare
Owner
|
@ntdatt812 thanks, merged 👍 Appreciate that you came back and found the one write #3056 didn't cover. And you tested it on real cp1252 streams, not mocks. If you feel like doing another, #3246 is open. 🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of the same shape as #3050 (startup banners) and #3056 (result lines, and the spinner). The one terminal write in the AI module that was not covered is the analysis itself.
The defect
_stream_response()writes each token of the model's reply straight to stdout:The reply is arbitrary text. An OSINT run on an international username routinely comes back with Cyrillic or CJK. On Windows, Python takes stdout's encoding from the process ANSI codepage — cp1252 on a default install — so that write raises
UnicodeEncodeError, the exception leaves the coroutine, and the analysis ends mid-sentence with the tokens already printed left on screen.PYTHONIOENCODINGcannot rescue the PyInstaller build, which ignoresPYTHON*variables.surrogateescapeis not a guard here, which is the part worth pinning. It is whatsys.stdoutactually carries, and it rescues lone surrogates only — an ordinary unencodable character still raises:test_raw_write_to_a_cp1252_stdout_is_what_failsasserts that, so the suite notices if the platform ever changes underneath the fix. The fix itself reuses the shape from the two merged PRs —notify._print_encodable— as a stream write rather than a print.A correction to something I wrote
The
_frames_fordocstring, which I added in #3056, says the braille frames "raised UnicodeEncodeError inside the spinner's daemon thread: the thread died with a traceback over the output".They did not. I only found this while probing upstream
mainfor this PR. Unlike stdout,sys.stderr's error handler isbackslashreplace, andPYTHONIOENCODINGdoes not change it:sys.stdoutsurrogateescapesys.stderrbackslashreplace⠋So the spinner never crashed — the animation just became a column of literal escapes. The fix was right; the reason I gave for it was not, and the difference between the two streams is exactly what this change turns on. The docstring now says what actually happens.
Left alone
print_streaming()writes to stdout the same way and has the same defect, but it has no callers anywhere in the tree. I would rather not change code there is no way to exercise; happy to include it if you would prefer.Verification
6 tests added to
tests/test_ai.py, driven by realcp1252TextIOWrapperstreams rather than mocks, so the codec decides. Checked by breaking it:test_streaming_an_international_reply_finishes..._does_not_raisecases + the end-to-end onetest_a_stream_that_does_not_raise_is_left_to_its_own_handlerThe last test exists because that third mutation survived my first draft: every other case here uses text cp1252 either carries unchanged or cannot carry at all, and both paths give the same bytes for those. A
backslashreplacestream is the one that separates them.Full suite on Windows: the set of failing test names is identical before and after (35 pre-existing, plus
tests/test_maigret.py, which cannot be collected on Windows at all —os.geteuiddoes not exist there).flake8clean.