Skip to content

Commit 2c8cd3a

Browse files
authored
Add FAQ entry about UnicodeEncodeError on Windows (#3778)
2 parents e1fd594 + 131c86a commit 2c8cd3a

1 file changed

Lines changed: 45 additions & 0 deletions

File tree

‎docs/faqs.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,48 @@ Note that I used single quotes above, so my shell is not expanding the environme
3737

3838
If you don't want Click to emulate (as best it can) unix expansion on Windows, pass windows_expand_args=False when calling the CLI.
3939
Windows command line doesn't do any *, ~, or $ENV expansion. It also doesn't distinguish between double quotes and single quotes (where the later means "don't expand here"). Click emulates the expansion so that the app behaves similarly on both platforms, but doesn't receive information about what quotes were used.
40+
41+
### `UnicodeEncodeError` on Windows
42+
43+
A Click CLI that uses Unicode characters in its help text or `echo` output can raise `UnicodeEncodeError`:
44+
45+
```pytb
46+
UnicodeEncodeError: 'charmap' codec can't encode character '\u2023' in position 62: character maps to <undefined>
47+
```
48+
49+
This is not a bug in Click. When `<stdout>` is a console, Click relies on the Windows console API, which supports full Unicode rendering. But when `<stdout>` is redirected, it cannot use that API, so Python falls back to encoding the stream with the system's code page (often `cp1252`) and `errors="strict"`. That codec cannot represent every Unicode character, so the write fails.
50+
51+
#### Answer
52+
53+
Run Python in [UTF-8 mode](https://docs.python.org/3/library/os.html#utf8-mode) so that redirected streams use UTF-8 instead of the code page.
54+
55+
You can enforce that by:
56+
- Setting the `PYTHONUTF8=1` environment variable, or
57+
- Passing `-X utf8` to the `python` executable, or
58+
- Setting `PYTHONIOENCODING=utf8`, which configures only the standard streams, or
59+
- From your code, call `sys.stdout.reconfigure(encoding="utf-8")` and `sys.stderr.reconfigure(encoding="utf-8")` before producing output.
60+
61+
This is not a bug for Click to fix, but a system configuration. It will naturally go away with UTF-8 mode becoming the default in Python 3.15 (see [PEP 686](https://peps.python.org/pep-0686/)).
62+
63+
#### For library authors
64+
65+
Click intentionally does not fix this in code: it has no way to know what the caller expects. Reconfiguring the stream itself would mix encodings within one process: `echo` could write UTF-8 to a redirected file while a bare `print()` still writes the code page. A loud `UnicodeEncodeError` is easier to diagnose than silently interleaved encodings.
66+
67+
But if your own CLI knows the environment it runs under, it can choose to degrade its output:
68+
69+
```python
70+
import sys
71+
72+
encoding = getattr(sys.stdout, "encoding", "") or ""
73+
ascii_only = not encoding.lower().startswith("utf")
74+
```
75+
76+
Note that the check we recommend here is against `sys.stdout.encoding`, not `sys.flags.utf8_mode`: only the former reports UTF-8 for all three options listed below.
77+
78+
| Invocation | `sys.flags.utf8_mode` | `sys.stdout.encoding` |
79+
| :-------------------------------------- | :-------------------: | :-------------------: |
80+
| `PYTHONIOENCODING=utf8` | `0` | `utf-8` |
81+
| `python -X utf8` | `1` | `utf-8` |
82+
| `sys.stdout.reconfigure(encoding=…)` | `0` | `utf-8` |
83+
84+
Do not rely on `sys.getfilesystemencoding()` on Windows either: [PEP 529](https://peps.python.org/pep-0529/) makes it `utf-8` regardless of the actual stream encoding.

0 commit comments

Comments
 (0)