Repository navigation
fix(workflows): decode code-page shell output in Python's UTF-8 mode - #5209
Merged
Merged
Conversation
The shell step decodes captured output with locale.getpreferredencoding(False), as text=True would. In Python's UTF-8 mode (PYTHONUTF8=1, -X utf8, and the default from Python 3.15) that is always "utf-8", but cmd.exe and other native Windows programs still write in the ANSI code page. On a cp932 or cp936 system a step running `echo テスト` then fails with "stdout could not be decoded". When the output is not valid in the preferred encoding, retry with locale.getencoding(), the code page text=True used outside UTF-8 mode. Output that is invalid in both still fails the step with the original error. Assisted-by: Claude Code (model: Claude Opus 5.5, autonomous) Co-Authored-By: Claude Opus 5.5 <[email protected]>
mnriem
added
the
triage-must-have
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the shell-output decoding fallback.
No open issues 🎉
What changed in this PR
This PR fixes shell-step output decoding when Python’s UTF-8 mode differs from the native system code page, while preserving structured failures for undecodable output.
Changes:
- Retry failed decoding with the locale encoding when it differs from the preferred encoding.
- Preserve universal-newline handling and the original decoding error if both attempts fail.
- Add regression coverage for code-page text and JSON output, plus invalid-byte failure coverage.
| File | Description |
|---|---|
tests/workflows/test_shell_step.py |
Tests code-page decoding in simulated UTF-8 mode and failure when neither encoding works. |
src/specify_cli/workflows/step/shell/__init__.py |
Adds locale-encoding fallback for captured stdout and stderr. |
Review effort: Balanced (auto)
Collaborator
|
Thank you! |
kevinnguyenhoang91
pushed a commit
to kevinnguyenhoang91/spec-kit
that referenced
this pull request
Nov 10, 2026
…ithub#5209) The shell step decodes captured output with locale.getpreferredencoding(False), as text=True would. In Python's UTF-8 mode (PYTHONUTF8=1, -X utf8, and the default from Python 3.15) that is always "utf-8", but cmd.exe and other native Windows programs still write in the ANSI code page. On a cp932 or cp936 system a step running `echo テスト` then fails with "stdout could not be decoded". When the output is not valid in the preferred encoding, retry with locale.getencoding(), the code page text=True used outside UTF-8 mode. Output that is invalid in both still fails the step with the original error. Assisted-by: Claude Code (model: Claude Opus 5.5, autonomous) Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The shell step decodes captured output with
locale.getpreferredencoding(False), astext=Truewould (#5064). In Python's UTF-8 mode (PYTHONUTF8=1,-X utf8, and the default from Python 3.15) that is always"utf-8". But cmd.exe and other native Windows programs still write in the ANSI code page, so on a cp932 or cp936 system any shell step that prints non-ASCII text fails.Reproduction on Windows (code page 932, Python 3.11), running
ShellStep().execute({"id": "s", "type": "shell", "run": "echo テスト"}, StepContext(project_root=".")):python -X utf8, onmain:FAILED,Shell command stdout could not be decoded: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start bytepythonwithout UTF-8 mode, onmain:COMPLETED, stdout'テスト\n'python -X utf8, with this change:COMPLETED, stdout'テスト\n'With this change, output that is not valid in the preferred encoding is decoded again with
locale.getencoding(), the code pagetext=Trueused outside UTF-8 mode. Nothing changes when the two encodings are the same (outside UTF-8 mode, or on a UTF-8 locale). Output that is invalid in both still fails the step with the original error, so #5064's behavior is kept.Testing
New tests in
tests/workflows/test_shell_step.py, run in a fresh Python 3.11 venv on Windows:test_code_page_output_is_decoded_in_utf8_mode[text]and[json]simulate UTF-8 mode on a cp936 system with a child process that writes cp936 bytes. Both fail onmain(assert StepStatus.FAILED == StepStatus.COMPLETED) and pass with the change.test_output_invalid_in_code_page_too_fails_step(negative case) checks that bytes invalid in both encodings still fail the step with the UTF-8 error. It passes before and after.tests/workflows/test_shell_step.py: 73 passed, 21 skipped, both normally and under-X utf8.tests/workflowsandtests/test_workflows.py: 1412 passed, 28 skipped, 12 failed. The same 12 symlink tests fail onmainon this machine (no symlink privilege).ruff checkpasses on both changed files.Tested locally with
uv run specify --helpRan existing tests with
uv sync && uv run pytestTested with a sample project (if applicable)
AI Disclosure
AI disclosure: Claude Code with Claude Opus 5.5, running autonomously on behalf of @b84330808 with default settings. It wrote the code change, the tests, the commit message and this description, and ran the tests and the reproduction above. No human line-by-line review or manual testing is claimed. The commit has an
Assisted-by:trailer.I used Claude Code to write this change.