Skip to content

fix: keep the AI analysis from crashing on a non-UTF-8 stdout - #3096

Merged
soxoj merged 1 commit into
soxoj:mainfrom
ntdatt812:fix/ai-output-encoding
Oct 10, 2026
Merged

soxoj merged 1 commit into
soxoj:mainfrom
ntdatt812:fix/ai-output-encoding

Conversation

@ntdatt812

@ntdatt812 ntdatt812 commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Third of the same shape as #3050 (startup banners) and #3056 (result lines, and the spinner). The one terminal write in the AI module that was not covered is the analysis itself.

The defect

_stream_response() writes each token of the model's reply straight to stdout:

sys.stdout.write(content)
sys.stdout.flush()

The reply is arbitrary text. An OSINT run on an international username routinely comes back with Cyrillic or CJK. On Windows, Python takes stdout's encoding from the process ANSI codepage — cp1252 on a default install — so that write raises UnicodeEncodeError, the exception leaves the coroutine, and the analysis ends mid-sentence with the tokens already printed left on screen. PYTHONIOENCODING cannot rescue the PyInstaller build, which ignores PYTHON* variables.

surrogateescape is not a guard here, which is the part worth pinning. It is what sys.stdout actually carries, and it rescues lone surrogates only — an ordinary unencodable character still raises:

>>> "Привет".encode("cp1252", "surrogateescape")
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-5

test_raw_write_to_a_cp1252_stdout_is_what_fails asserts that, so the suite notices if the platform ever changes underneath the fix. The fix itself reuses the shape from the two merged PRs — notify._print_encodable — as a stream write rather than a print.

A correction to something I wrote

The _frames_for docstring, which I added in #3056, says the braille frames "raised UnicodeEncodeError inside the spinner's daemon thread: the thread died with a traceback over the output".

They did not. I only found this while probing upstream main for this PR. Unlike stdout, sys.stderr's error handler is backslashreplace, and PYTHONIOENCODING does not change it:

stream encoding errors a braille frame
sys.stdout cp1252 surrogateescape raises
sys.stderr cp1252 backslashreplace prints ⠋

So the spinner never crashed — the animation just became a column of literal escapes. The fix was right; the reason I gave for it was not, and the difference between the two streams is exactly what this change turns on. The docstring now says what actually happens.

Left alone

print_streaming() writes to stdout the same way and has the same defect, but it has no callers anywhere in the tree. I would rather not change code there is no way to exercise; happy to include it if you would prefer.

Verification

6 tests added to tests/test_ai.py, driven by real cp1252 TextIOWrapper streams rather than mocks, so the codec decides. Checked by breaking it:

reverted killed
stream the reply with a raw write again test_streaming_an_international_reply_finishes
drop the helper's fallback, keep the call site both ..._does_not_raise cases + the end-to-end one
always replace, never write straight through test_a_stream_that_does_not_raise_is_left_to_its_own_handler

The last test exists because that third mutation survived my first draft: every other case here uses text cp1252 either carries unchanged or cannot carry at all, and both paths give the same bytes for those. A backslashreplace stream is the one that separates them.

Full suite on Windows: the set of failing test names is identical before and after (35 pre-existing, plus tests/test_maigret.py, which cannot be collected on Windows at all — os.geteuid does not exist there). flake8 clean.

Third of the same shape as soxoj#3050 (startup banners) and soxoj#3056 (result
lines and the spinner). The one terminal write in the AI module that was
not covered is the analysis itself.

_stream_response() writes each token of the model's reply straight to
stdout. The reply is arbitrary text, and an OSINT run on an international
username routinely comes back with Cyrillic or CJK. On Windows, Python
takes stdout's encoding from the process ANSI codepage -- cp1252 on a
default install -- so that write raises UnicodeEncodeError, the exception
leaves the coroutine, and the analysis ends mid-sentence with the tokens
already printed left on screen.

sys.stdout's error handler is surrogateescape, which is not a guard here:
it rescues lone surrogates only, so an ordinary unencodable character
still raises. There is a test that pins exactly that, so this suite
notices if the platform ever changes.

Also corrects the _frames_for docstring, which I wrote in soxoj#3056. It says
the braille frames "raised UnicodeEncodeError inside the spinner's daemon
thread". They did not: unlike stdout, stderr's error handler is
backslashreplace, so the write never raised -- it printed the escape, and
the animation became a column of literal ⠋. The fix was right, the
reason given for it was not, and the difference between the two streams
is exactly what this change turns on.

print_streaming() writes to stdout the same way and has the same defect,
but it has no callers anywhere in the tree, so it is left alone rather
than changed without a way to exercise it.

Verified: 6 tests added to tests/test_ai.py, driven by real cp1252 streams
rather than mocks so the codec decides. Three mutations, all killed.
Full suite on Windows: the set of failing test names is identical before
and after (35 pre-existing).
@ntdatt812
ntdatt812 force-pushed the fix/ai-output-encoding branch from 5d2d7b0 to 613c419 Compare September 7, 2026 09:06
@soxoj
soxoj merged commit bd21950 into soxoj:main Oct 10, 2026
6 checks passed
@soxoj

soxoj commented Oct 10, 2026

Copy link
Copy Markdown
Owner

@ntdatt812 thanks, merged 👍

Appreciate that you came back and found the one write #3056 didn't cover. And you tested it on real cp1252 streams, not mocks.
That's your third one here, it's good to see you keep coming back.

If you feel like doing another, #3246 is open. 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants